Strawberry weight estimation method based on multi-modal visual information

Abstract

The invention discloses a strawberry weight estimation method based on multi-modal visual information, and belongs to the technical field of computer vision and agricultural information. According to the method, 2D RGB visual information, depth information, three-dimensional point cloud geometric features and color features are fused, a multi-modal information fusion framework is constructed, and interference caused by strawberry shape irregularity, partial shielding, posture variability and color difference is resisted; by analyzing the local neighborhood relation of the three-dimensional point cloud, the actual physical size represented by each pixel is dynamically calculated to obtain a dynamic conversion factor, and the change of the self-adaptive shooting distance and the influence of the posture of the strawberry are obtained; in combination with a plurality of key features of strawberries, a multi-modal comprehensive feature vector is constructed, so that a regression model can learn a deeper and more complex mapping relationship between the weight and the strawberry appearance. According to the method, the limitation that a traditional method is high in dependence on key point depth and poor in robustness is overcome, and non-contact, automatic and high-robustness weight detection is achieved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and agricultural information technology, and in particular to a method for estimating the weight of strawberries based on multimodal visual information. Background Technology

[0002] With the development of smart agriculture and automated post-harvest processing technologies for fruits and vegetables, rapid, accurate, and non-contact weight grading and estimation of fruits has become a key link in improving industry efficiency. Traditional weight measurement methods mainly rely on mechanical weighing, which is inefficient and prone to damaging fruits and vegetables, making it difficult to meet the high-throughput requirements of automated production lines.

[0003] In recent years, non-contact weight estimation methods based on machine vision have attracted widespread attention. Existing technologies mainly rely on a single source of visual information, such as methods based on 2D RGB images. These typically extract morphological features like area, perimeter, and diameter from the 2D projected image of the fruit, or utilize features such as color histograms, estimating weight by establishing a regression model between features and weight. However, fruits and vegetables like strawberries have irregular shapes and varied postures, resulting in a lack of stable mapping between their 2D projected area and actual volume / weight. Furthermore, occlusion in the image, changes in lighting, and the diversity of fruit colors (such as ripeness differences) can severely interfere with the accuracy of feature extraction. Another approach is based on depth images or 3D point clouds, using depth cameras or 3D reconstruction techniques to acquire point cloud data of fruits and vegetables, estimating weight by calculating the point cloud volume or bounding box size. These methods are closer to the physical essence than two-dimensional methods, but calculating the physical distance between two points directly from depth images or point clouds is limited by the noise of depth sensors and the perspective projection distortion of cameras, resulting in low measurement accuracy. At the same time, single geometric volume features often ignore the information on fruit density, internal structure (such as hollowness), and maturity (related to density) implied by surface color, leading to poor universality of the estimation model.

[0004] In summary, existing non-contact fruit and vegetable weight estimation methods have the following drawbacks: First, the information dimension is limited, resulting in insufficient estimation accuracy and robustness in complex scenarios such as irregular strawberry shapes, occlusion, and color variations. Second, the physical scale conversion is inaccurate, with errors in the conversion process from image pixels to real-world physical dimensions, directly affecting the accuracy of subsequent geometric feature calculations. Third, feature representation is incomplete, as feature vectors fail to systematically integrate multi-dimensional physical attributes such as morphology, geometry, and color, limiting the machine learning model's ability to learn the complex nonlinear relationship between weight and appearance.

[0005] Current technologies do not yet provide a solution for stably obtaining physical scales even when keypoint depth information is missing, noise is severe, or pose changes are significant, and for improving weight estimation accuracy by combining multi-physical dimension features. Therefore, there is an urgent need for a highly robust and accurate visual multimodal strawberry weight estimation method. Summary of the Invention

[0006] To address the problem that traditional methods rely on keypoint depth and lack robustness, this invention provides a strawberry weight estimation method based on multimodal visual information. This method can still robustly calculate the true physical size when keypoint depth fails, and achieves high-precision weight prediction through multimodal feature fusion.

[0007] This invention adopts the following technical solution: a strawberry weight estimation method based on multimodal visual information, comprising the following steps:

[0008] Step 1, Image Acquisition: The camera is pre-calibrated, and RGB and depth images of strawberries are acquired simultaneously based on the principle of binocular parallax, and the camera intrinsic parameter matrix is ​​obtained;

[0009] Step 2, Keypoint Detection: Input the RGB image of the strawberry into the pre-trained YOLOv11n-pose model, and output the pixel coordinates of four key points of the strawberry (top, bottom, left, and right) to obtain the keypoint set;

[0010] Step 3, Segmentation Mask Acquisition and Depth Information Extraction: Input the RGB image into the pre-trained YOLOv11m-seg model to generate the segmentation mask of the strawberry instance, obtain the depth map of the strawberry region, and perform statistical filtering processing;

[0011] Step 4, Point Cloud Generation: Using the camera intrinsic parameter matrix, project each filtered pixel-depth pair back onto the 3D camera coordinate system to obtain the 3D point cloud of the strawberry.

[0012] Step 5: Principal component feature extraction: Perform PCA analysis on the 3D point cloud to obtain its geometric orientation and extract principal component features;

[0013] Step 6: Calculate the conversion factor: Calculate the average physical distance between adjacent points in the point cloud, correlate it with the number of pixels, and obtain the conversion factor. , which serves as a scaling factor from pixel distance to physical distance;

[0014] Step 7, Physical Size Calculation: Calculate the physical size of the strawberry using key points and conversion factors;

[0015] Step 8: Maturity Calculation: Estimate the ripeness of strawberries by measuring the proportion of the red channel in the RGB color space;

[0016] Step 9, Comprehensive Feature Construction and Weight Estimation: Construct a comprehensive feature vector, input it into the pre-trained random forest regression model, and output the final weight estimate by integrating the prediction results of multiple decision trees.

[0017] Preferably, the RGB image is represented as The depth image is represented as ;in, These are pixel coordinates;

[0018] The intrinsic parameter matrix is ​​represented as follows: :

[0019] ;

[0020] in, These are the focal lengths in the x and y directions, respectively. These are the principal coordinates of the optical center.

[0021] Preferably, the YOLOv11n-pose model described in step 2 outputs the pixel coordinates of four key points of the strawberry. This yields the set of key points, represented as .

[0022] Preferably, the segmentation mask acquisition and depth information extraction in step 3 are performed as follows:

[0023] Step 3.1: The YOLOv11m-seg model generates the segmentation mask for the strawberry instance. Obtain the depth map of the strawberry area. This restricts depth information to the strawberry region:

[0024] ;

[0025] Among them, the segmentation mask It is a binary image, with the strawberry region set to 1 and the background set to 0.

[0026] Step 3.2: Depth map of the strawberry area The set of all valid depth values ​​in Perform statistical filtering, including: calculating the mean. and standard deviation And remove outliers, retaining only those within the interval. The depth value inside, The scale parameter forms the filtered depth set. .

[0027] Preferably, step 4 involves back-projecting each filtered pixel-depth pair onto the 3D camera coordinate system to obtain the 3D point cloud of the strawberry. , is represented as: ;in, This represents the coordinates of the j-th point cloud. .

[0028] Preferably, the principal component feature extraction in step 5 is performed as follows:

[0029] Step 5.1: Calculate the centroid of the point cloud. :

[0030] ;

[0031] Step 5.2: Calculate the covariance matrix :

[0032] ;

[0033] Step 5.3: Perform feature decomposition:

[0034] ;

[0035] in, It is an eigenvalue. It is the corresponding feature vector;

[0036] Step 5.4: Extract principal component features: Take the first two feature vectors. This serves as the principal component feature describing the main spatial distribution direction of strawberries.

[0037] Preferably, the conversion factor calculation in step 6 is performed as follows:

[0038] Step 6.1: Calculate the physical average distance Point cloud calculation using KD-Tree data structure Calculate the distance between each point and its nearest neighbor, and then average the distances.

[0039] The physical average distance It is the average physical distance of one step in three-dimensional space, corresponding to pixel-level changes in a two-dimensional image;

[0040] Step 6.2: Obtain the conversion factor , which serves as a scaling factor from pixel distance to physical distance.

[0041] Preferably, the physical dimension calculation in step 7 is performed as follows:

[0042] Step 7.1, pixel distance calculation, including:

[0043] Calculate the horizontal pixel distance: ;

[0044] Calculate the pixel distance along the vertical axis: ;

[0045] Step 7.2, Physical dimension calculation, including:

[0046] Calculate the physical length of the horizontal axis: ;

[0047] Calculate the physical length of the vertical axis: .

[0048] Preferably, the maturity calculation in step 8 is performed as follows:

[0049] Step 8.1: Extract the RGB region of the strawberry. :

[0050] ;

[0051] Step 8.2, Count red pixels: Set a red threshold. Statistical satisfaction and and Number of pixels ;

[0052] in, , , These represent the red value, green value, and blue value of a pixel, respectively.

[0053] Step 8.3: Calculate maturity level :

[0054] ;

[0055] in, It is a segmentation mask Total number of effective pixels.

[0056] Preferably, the comprehensive feature construction and weight estimation in step 9 are performed as follows:

[0057] Step 9.1: Construct a comprehensive feature vector :

[0058] ;

[0059] in: This represents the pixel area of ​​the segmentation mask. Represents the features of the depth histogram. This represents the principal component eigenvectors obtained by PCA. Indicates maturity level. Indicates physical dimensions;

[0060] Step 9.2: Combine the feature vectors Input a pre-trained random forest regression model :

[0061] ;

[0062] Step 9.3: The random forest model outputs the final weight estimate by integrating the prediction results of multiple decision trees. .

[0063] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0064] 1. Multimodal information fusion framework for fruit and vegetable weight estimation: This invention proposes a multimodal information fusion framework that integrates RGB 2D visual information, depth information, 3D point cloud geometric features and color features. It effectively combats interference caused by irregular strawberry shape, partial occlusion, varied posture and color differences. It breaks through the limitations of traditional reliance on only two-dimensional images or a single data source. Through information complementarity, it greatly improves the accuracy and robustness of non-contact weight estimation.

[0065] 2. Precise Measurement of Physical Size Based on Dynamic Transformation Factor: This invention proposes the concept of transformation factor and its dynamic calculation method. By analyzing the local neighborhood relationship of the 3D point cloud, the actual physical size represented by each pixel is dynamically calculated. The transformation factor can adapt to changes in shooting distance and the influence of the strawberry's own posture, effectively suppressing errors caused by depth image noise and camera perspective projection distortion. This fundamentally ensures the accuracy of scale conversion from the pixel coordinate system to the physical world coordinate system, laying a reliable foundation for subsequent feature calculation based on physical size.

[0066] 3. Construction of Multi-Dimensional Feature Vectors for Weight Estimation: This invention constructs a comprehensive multi-dimensional feature vector that integrates three major categories of information: morphology, geometry, and color. It includes not only morphological features based on precise physical dimensions (such as length and width), but also geometric features describing the spatial orientation and surface undulation details of the fruit (such as principal component features and depth histograms), as well as color features reflecting maturity information (such as the proportion of red pixels). This provides a comprehensive digital description of strawberries from multiple physical dimensions, with rich information content and clear physical meaning. This enables subsequent regression models to learn a deeper and more complex mapping relationship between weight and fruit and vegetable appearance, significantly improving the model's generalization ability and estimation accuracy.

[0067] 4. Non-contact detection and automated grading: The method of this invention is entirely based on visual information, requiring no contact with the fruit and avoiding mechanical damage that may be caused by traditional weighing methods. The entire process can be automated, is fast, and meets the high-throughput and real-time requirements of modern fruit and vegetable post-harvest processing lines, showing good prospects for industrial application. Attached Figure Description

[0068] Figure 1 This is a flowchart of the strawberry weight estimation method based on multimodal visual information of the present invention;

[0069] Figure 2 This is a schematic diagram of key point detection in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0071] In one embodiment of the present invention, a strawberry weight estimation method based on multimodal visual information is provided, such as... Figure 1 As shown, the steps are as follows:

[0072] 1. Image Acquisition

[0073] Using a D405 RGB-D camera, RGB images of strawberries were simultaneously acquired through the principle of binocular parallax. and depth images ,in, These are pixel coordinates.

[0074] The camera needs to be calibrated in advance to obtain the camera intrinsic parameter matrix. :

[0075] ;

[0076] in, These are the focal lengths in the x and y directions, respectively (in pixels). These are the principal coordinates of the optical center.

[0077] 2. Keypoint Detection

[0078] The RGB image obtained in step 1 Input a pre-trained YOLOv11n-pose model, and the model outputs the pixel coordinates of four key points of a strawberry. , specifically Figure 2 As shown.

[0079] Further obtain the key point set : ;

[0080] in, , .

[0081] 3. Segmentation mask acquisition and depth information extraction

[0082] First, convert the RGB image Input the pre-trained YOLOv11m-seg model to generate segmentation masks for strawberry instances. This is a binary image where the strawberry region is 1 and the background is 0.

[0083] Then, the depth map of the strawberry area is obtained. :

[0084] ;

[0085] This operation restricts depth information to the strawberry area.

[0086] Furthermore, depth information processing is performed based on the obtained depth map of the strawberry area: The set of all valid depth values Perform statistical filtering.

[0087] Calculate the mean and standard deviation :

[0088] ;

[0089] ;

[0090] in, Indicates the first One effective depth value, This represents the number of valid pixels in the strawberry region within the neighborhood.

[0091] Outlier removal: retaining points within the interval The depth values ​​within the range form a filtered depth set. .

[0092] in, This indicates the number of valid points in the 3D point cloud of the strawberry plant. This is a scale parameter, which is set to 2.0 or 3.0 in this embodiment.

[0093] 4. Point Cloud Generation

[0094] Using the camera intrinsic parameter matrix obtained in step 1 Each pixel-depth pair after filtering Back-projected onto the 3D camera coordinate system.

[0095] The back projection formula is as follows:

[0096] ;

[0097] This formula can be used to obtain the 3D point cloud of a strawberry. ,in .

[0098] 5. PCA Analysis and Principal Component Extraction

[0099] The point cloud obtained in step 4 Principal component analysis is performed to obtain its geometric orientation, as follows:

[0100] Calculate the centroid of the point cloud :

[0101] ;

[0102] Calculate the covariance matrix :

[0103] ;

[0104] Eigenvalue decomposition:

[0105] ;

[0106] in It is an eigenvalue. It is the corresponding feature vector.

[0107] Principal component features: In this embodiment, the first two feature vectors are taken. This serves as the principal component feature describing the main spatial distribution direction of strawberries.

[0108] 6. Conversion factor calculation

[0109] Calculate the point cloud obtained in step 4 The average physical distance between adjacent points is calculated and correlated with the number of pixels to obtain the physical size represented by each pixel.

[0110] Specifically, physical average distance Efficiently compute point clouds using data structures such as KD-Tree The distance between each point and its nearest neighbor is calculated, and then the average is obtained using the following formula:

[0111] ;

[0112] in, yes The nearest neighbor.

[0113] Then, the conversion factor is obtained. :

[0114] ;

[0115] Here, each pixel is implicitly included in the calculation of the physical average distance; more accurately, It is the average physical distance of "one step" in three-dimensional space, which corresponds to pixel-level changes in a two-dimensional image. Therefore, It is used directly as a scaling factor from pixel distance to physical distance.

[0116] It is particularly important to note that traditional object size measurement methods based on depth cameras typically rely on directly calculating the 3D Euclidean distance between specific feature points. However, in real-world agricultural scenarios, strawberry surfaces often suffer from missing or unreliable local depth information due to factors such as reflective properties, irregular shapes, or occlusion. In particular, key feature points on strawberries are often located in areas where depth information is most vulnerable: the top surface is prone to specular reflection due to its smoothness, while the bottom surface is weakened by the dark color of the sepals, which absorb light. When the depth values ​​of key points are invalid, traditional direct calculation methods will completely fail.

[0117] Based on this, the transformation factor method proposed in this invention fundamentally solves this problem through innovative mathematical modeling. The core idea of ​​this method is to establish a robust mapping relationship between the image pixel coordinate system and the physical space coordinate system, rather than relying on the precise depth value of a single point. Specifically, the transformation factor... The average physical displacement corresponding to a unit pixel displacement was obtained through statistical analysis of a large number of effective depth points on the strawberry surface.

[0118] This design offers three advantages:

[0119] First, the measurement process is immune to local depth loss. Even if the depth information at key points is completely missing, as long as the pixel coordinates of the key points can be accurately located from the RGB image, the physical size can be obtained through a simple calculation of "pixel distance × conversion factor". The calculation process completely avoids direct dependence on the depth value of the problem area.

[0120] Secondly, it improves the statistical stability of the measurement. (Transformation factor) Based on statistical calculations using hundreds to thousands of valid depth points, the impact of a single or small number of outliers on the results is limited. This collective intelligence approach is more robust than methods that rely on depth values ​​from a few key points and can effectively suppress random noise and local anomalies in depth images.

[0121] Finally, the synergistic advantages of multimodal data are leveraged. This method cleverly separates the strengths of different data sources: utilizing the high accuracy of RGB images in texture recognition and edge detection ensures the accuracy of keypoint pixel localization; leveraging the spatial geometric perception capabilities of depth information establishes scale relationships through statistical methods. These complementary approaches allow the system to maintain functionality even when depth information is incomplete.

[0122] 7. Physical Dimension Calculation

[0123] Using the four key points in step 2 and the conversion factor obtained in step 6 The specific method for calculating the physical dimensions of strawberries is as follows:

[0124] First, calculate the pixel distance, including:

[0125] Horizontal axis pixel distance: ;

[0126] Vertical axis pixel distance: ;

[0127] Then, calculate the physical dimensions, including:

[0128] Horizontal axis physical length: ;

[0129] Vertical axis physical length: .

[0130] 8. Maturity Calculation

[0131] In the RGB color space, maturity is estimated by the proportion of the red channel. The specific method is as follows:

[0132] First, extract the RGB region of the strawberry:

[0133]

[0134] Then, count the red pixels and set a red threshold. (For example ), statistical satisfaction and and Number of pixels .

[0135] Next, calculate the ripeness of the strawberries. :

[0136]

[0137] in, It is a segmentation mask The total number of effective pixels (i.e., the total number of pixels in the strawberry).

[0138] 9. Comprehensive Feature Construction and Weight Estimation

[0139] The multimodal features extracted in the previous steps are further constructed into a comprehensive feature vector, which is then input into the regression model.

[0140] Specifically, feature vector Represented as: ;

[0141] in, , representing the pixel area of ​​the segmentation mask, as a supplement to the two-dimensional size; Representing depth histogram features; dividing the filtered depth value range into... Each interval (bin) is counted to determine the number of points in each interval, forming a... dimensional vector; This represents the principal component eigenvectors obtained by PCA (a total of 6 scalars); Indicates maturity level; Indicates physical dimensions.

[0142] As a preferred embodiment, in this example, .

[0143] Then, weight estimation is performed, and the eigenvectors are... Input a pre-trained random forest regression model :

[0144] ;

[0145] Finally, the random forest model outputs the final weight estimate by integrating the predictions from multiple decision trees (e.g., by averaging). .

[0146] The strawberry weight prediction results of the random forest model in this embodiment are shown in Table 1 below:

[0147] Table 1 strawberry samples Predicted weight (g) Actual weight (g) Absolute error (g) Relative error (%) #1 24.6 25.2 -0.6 2.4% #2 32.1 31.5 +0.6 1.9% #3 18.9 19.3 -0.4 2.1% #4 28.3 27.8 +0.5 1.8% #5 35.7 36.5 -0.8 2.2%

[0148] The performance statistics of this implementation are as follows: the mean absolute error (MAE) is 0.58g, the mean relative error is 2.1%, and the coefficient of determination (R²) is 0.974. It is evident that the method of this invention greatly improves the accuracy and robustness of non-contact weight estimation.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for estimating the weight of strawberries based on multimodal visual information, characterized in that, Includes the following steps: Step 1, Image Acquisition: The camera is pre-calibrated, and RGB and depth images of strawberries are acquired simultaneously based on the principle of binocular parallax, and the camera intrinsic parameter matrix is ​​obtained; Step 2, Keypoint Detection: Input the RGB image of the strawberry into the pre-trained YOLOv11n-pose model, and output the pixel coordinates of four key points of the strawberry (top, bottom, left, and right) to obtain the keypoint set; Step 3, Segmentation Mask Acquisition and Depth Information Extraction: Input the RGB image into the pre-trained YOLOv11m-seg model to generate the segmentation mask of the strawberry instance, obtain the depth map of the strawberry region, and perform statistical filtering processing; Step 4, Point Cloud Generation: Using the camera intrinsic parameter matrix, project each filtered pixel-depth pair back onto the 3D camera coordinate system to obtain the 3D point cloud of the strawberry. Step 5: Principal component feature extraction: Perform PCA analysis on the 3D point cloud to obtain its geometric orientation and extract principal component features; Step 6: Calculate the conversion factor: Calculate the average physical distance between adjacent points in the point cloud, correlate it with the number of pixels, and obtain the conversion factor. , which serves as a scaling factor from pixel distance to physical distance; Step 7, Physical Size Calculation: Calculate the physical size of the strawberry using key points and conversion factors; Step 8: Maturity Calculation: Estimate the ripeness of strawberries by measuring the proportion of the red channel in the RGB color space; Step 9, Comprehensive Feature Construction and Weight Estimation: Construct a comprehensive feature vector, input it into the pre-trained random forest regression model, and output the final weight estimate by integrating the prediction results of multiple decision trees.

2. The strawberry weight estimation method and system based on multimodal visual information according to claim 1, characterized in that, The RGB image is represented as The depth image is represented as ;in, These are pixel coordinates; The intrinsic parameter matrix is ​​represented as follows: : ； in, These are the focal lengths in the x and y directions, respectively. These are the principal coordinates of the optical center.

3. The strawberry weight estimation method and system based on multimodal visual information according to claim 2, characterized in that, The YOLOv11n-pose model described in step 2 outputs the pixel coordinates of the four key points of the strawberry. This yields the set of key points, represented as .

4. The strawberry weight estimation method and system based on multimodal visual information according to claim 2, characterized in that, Step 3, which involves obtaining the segmentation mask and extracting depth information, is performed as follows: Step 3.1: The YOLOv11m-seg model generates the segmentation mask for the strawberry instance. Obtain the depth map of the strawberry area. This restricts depth information to the strawberry region: ； Among them, the segmentation mask It is a binary image, with the strawberry region set to 1 and the background set to 0; Step 3.2: Depth map of the strawberry area The set of all valid depth values ​​in Perform statistical filtering, including: Calculate the mean and standard deviation : ； ； in, Indicates the first One effective depth value, This represents the number of valid pixels in the strawberry region within the neighborhood. Outlier removal: retaining points within the interval The depth value inside, The scale parameter forms the filtered depth set. , This indicates the number of valid points in the strawberry 3D point cloud.

5. The strawberry weight estimation method and system based on multimodal visual information according to claim 4, characterized in that, Step 4 involves back-projecting each filtered pixel-depth pair onto the 3D camera coordinate system, as shown in the following formula: ； Obtain the 3D point cloud of the strawberry , is represented as: ; in, For pixel-depth pairs, For pixels The corresponding depth, This represents the coordinates of the j-th point cloud. .

6. The strawberry weight estimation method and system based on multimodal visual information according to claim 5, characterized in that, The principal component feature extraction described in step 5 is performed as follows: Step 5.1: Calculate the centroid of the point cloud. : ； Step 5.2: Calculate the covariance matrix : ； Step 5.3: Perform feature decomposition: ； in, It is an eigenvalue. It is the corresponding feature vector; Step 5.4: Extract principal component features: Take the first two feature vectors. This serves as the principal component feature describing the main spatial distribution direction of strawberries.

7. The strawberry weight estimation method and system based on multimodal visual information according to claim 5, characterized in that, The conversion factor calculation in step 6 is performed as follows: Step 6.1: Calculate the physical average distance Point cloud calculation using KD-Tree data structure Calculate the distance between each point and its nearest neighbor, and then average the distances. ； in, yes The nearest neighbor; the physical average distance It is the average physical distance of one step in three-dimensional space, corresponding to pixel-level changes in a two-dimensional image; Step 6.2: Obtain the conversion factor As a scaling factor from pixel distance to physical distance: 。 8. The strawberry weight estimation method and system based on multimodal visual information according to claim 7, characterized in that, The physical dimension calculation method described in step 7 is as follows: Step 7.1, pixel distance calculation, including: Calculate the horizontal pixel distance: ; Calculate the pixel distance along the vertical axis: ; Step 7.2, Physical dimension calculation, including: Calculate the physical length of the horizontal axis: ; Calculate the physical length of the vertical axis: .

9. The strawberry weight estimation method and system based on multimodal visual information according to claim 7, characterized in that, The maturity calculation method described in step 8 is as follows: Step 8.1: Extract the RGB region of the strawberry. : ； Step 8.2, Count red pixels: Set a red threshold. Statistical satisfaction and and Number of pixels ; in, , , These represent the red value, green value, and blue value of a pixel, respectively. Step 8.3: Calculate maturity level : ； in, It is a segmentation mask Total number of effective pixels.

10. The strawberry weight estimation method and system based on multimodal visual information according to claim 9, characterized in that, The feature construction and weight estimation described in step 9 are performed as follows: Step 9.1: Construct a comprehensive feature vector : ； in: This represents the pixel area of ​​the segmentation mask. Represents the features of the depth histogram. This represents the principal component eigenvectors obtained by PCA. Indicates maturity level. Indicates physical dimensions; Step 9.2: Combine the feature vectors Input a pre-trained random forest regression model : ； Step 9.3: The random forest model outputs the final weight estimate by integrating the prediction results of multiple decision trees. .

Images