Vision-based tremella aurantialba intelligent positioning and maturity estimation method

Abstract

The invention discloses an intelligent tremella aurantialba positioning and maturity estimation method based on vision. The method comprises the following steps: acquiring a real-time image of tremella aurantialba; inputting the real-time image into a proportional relation model to obtain a corrected image; the corrected image is input into a tremella aurantialba positioning detection model, and a pixel-level mask and initial growth cycle classification are output; the corrected image and the output result of the tremella aurantialba positioning detection model are input into a tremella aurantialba height and diameter calculation model to obtain the height, diameter, positioning and form of the tremella aurantialba, and then the tremella aurantialba height, diameter, positioning and form are compared with a tremella aurantialba growth characteristic database to obtain the growth cycle of the tremella aurantialba. Through a double-camera system and a deep learning model, real-time monitoring and maturity judgment of the growth state of the tremella aurantialba are achieved, and a traditional mode depending on manual judgment is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a vision-based method for intelligent positioning and maturity estimation of golden ears. Background Art

[0002] In the field of agricultural automation, especially in the cultivation and harvesting of edible fungi, accurate detection and classification of fungi has always been the focus of the industry. Taking golden ear as an example, its growth process is divided into primordium stage, differentiation stage, color change stage and harvesting stage. The morphology and characteristics of each stage are different. The traditional manual judgment method is time-consuming and labor-intensive, and it is easy to miss or misdetect due to subjective factors.

[0003] In recent years, with the development of technologies such as computer vision and deep learning, the application of image processing in agriculture has become increasingly mature, providing new technical means for realizing automated monitoring and status assessment of edible fungi.

[0004] To solve

[0005] Due to the shortcomings of traditional manual monitoring, there is an urgent need to develop a vision-based intelligent positioning and maturity estimation system for golden ear. Summary of the invention

[0006] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the object of the present invention is to provide a vision-based method for intelligent positioning and maturity estimation of golden ear.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A vision-based method for intelligent positioning and maturity estimation of golden ear, comprising:

[0009] Get real-time images of golden ear;

[0010] The real-time image is input into the proportional relationship model to obtain a corrected image;

[0011] The corrected image is input into the golden ear positioning detection model, and a pixel-level mask and an initial growth cycle classification are output;

[0012] The corrected image and the output results of the golden ear positioning detection model are input into the golden ear height and diameter calculation model to obtain the height, diameter, positioning and shape of the golden ear, which are then compared with the golden ear growth feature database to obtain the growth cycle of the golden ear.

[0013] Furthermore, the real-time image of the golden ear is captured by dual cameras, which are perpendicular to each other and respectively capture a top view and a side view of the golden ear growing on the mushroom stick.

[0014] Furthermore, the proportional relationship model is used to correct camera distortion and achieve a preliminary conversion from pixels to actual size, specifically:

[0015] Using equally spaced concentric circles, cubic spline fitting is performed on the existing barrel distortion law of the image, and the corresponding coefficient matrix and fitting function are output;

[0016] Correct the image according to the output coefficient matrix and fitting function;

[0017] Determine the central cross section or vertical section of the mushroom stick as the reference plane, detect the pixel coordinate difference between the reference plane and the diameter or height of the reference object multiple times, calculate the ratio of the pixel coordinate to the actual coordinate, detect the pixel coordinate difference between the diameter or height of the reference object multiple times, and take the average value to obtain the ratio of the image pixel coordinate to the actual coordinate;

[0018] further,

[0019] The golden ear positioning detection model includes a feature extraction backbone network, a feature fusion network and a detection head;

[0020] Feature extraction backbone network: used to extract multi-layer feature maps of the rectified image, including multi-layer convolutional layers, layer-by-layer deep convolution, feature fusion and spatial pyramid;

[0021] Feature fusion network: used to obtain feature maps of multi-scale information of multi-layer feature maps, including up-sampling and down-sampling modules, feature fusion modules, splicing modules and multi-scale fusion modules;

[0022] Detection head: used to obtain pixel-level masks and initial maturity classification.

[0023] Furthermore, the golden ear positioning detection model adopts a combined loss function, which combines the bounding box overlap loss, the bounding box positioning loss and the segmentation loss.

[0024] Furthermore, the golden ear positioning detection model adopts a model training strategy of transfer learning.

[0025] Furthermore, the golden ear height and diameter calculation model specifically obtains the height and diameter characteristics of the golden ear through the corrected image, and then adjusts the height and diameter of the cap through the pixel ratio deviation, thereby obtaining the depth information of the golden ear from any camera, obtaining the coordinates of each key point in the golden ear contour in three-dimensional space and the morphological parameters of the cap, and obtaining the tangent vector and normal vector at its center of mass, thereby obtaining the actual height and diameter of the golden ear cap.

[0026] Furthermore, the growth cycle of golden ear includes primordium stage, opening stage, differentiation stage, color change stage and maturity stage.

[0027] Furthermore, a visualization step is included.

[0028] Furthermore, the golden ear growth characteristic database is constructed based on a large amount of golden ear growth data, and the golden ear growth data includes a reference range of golden ear diameter, height and color corresponding to the growth cycle of the golden ear.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] (1) The present invention uses a dual-camera system and a deep learning model to achieve real-time monitoring of the growth status and maturity judgment of golden ear fungus, avoiding the traditional method of relying on manual judgment. The system can automatically collect images and analyze the growth stage and maturity of golden ear fungus, greatly reducing the need for manual intervention and improving production efficiency.

[0031] (2) The present invention proposes an image-based golden ear growth relationship database. The database collects and analyzes a large amount of relevant data during the growth of golden ear, uses statistical and regression analysis methods, and establishes a mathematical relationship between the growth characteristics of golden ear and environmental conditions, providing a basis for size calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of a vision-based method and system for intelligent positioning and maturity estimation of golden ear of the present invention;

[0033] Figure 2 It is the structural flow chart of the golden ear positioning detection model;

[0034] Figure 3 It is the structural flow chart of the proportional relationship model;

[0035] Figure 4 It is a structural flow chart of the gold ear height and diameter calculation model;

[0036] FIG. 5( a ) and FIG. 5( b ) are actual pictures of the golden ear used in this embodiment and the obtained pixel-level mask. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below in conjunction with the examples, but the embodiments of the present invention are not limited thereto.

[0038] Example

[0039] like Figure 1-Figure 5(a) 5(b) shows a vision-based method for intelligent positioning and maturity estimation of golden ear. The method uses computer vision technology to intelligently detect, locate, calculate the size and determine the maturity of golden ear images, and controls the system environment through feedback to achieve automated monitoring and regulation of golden ear growth, including:

[0040] S1 obtains a real-time video of the golden ear. The golden ear is shown in FIG5(a). The real-time image is remotely obtained through dual cameras, specifically a side-view camera and a top-view camera. The former is used to capture the height characteristics of the golden ear, and the latter captures the diameter characteristics of the golden ear, which are used to assist in estimating its overall growth state and judging the maturity of the golden ear.

[0041] The maturity of the golden ear in this embodiment is the golden ear growth cycle, including the primordium stage, the opening stage, the differentiation stage, the color change stage and the maturity stage.

[0042] S2: the real-time image is input into the proportional relationship model to obtain a corrected image;

[0043] like Figure 3 As shown, the proportional relationship model is for the proportional relationship between the actual height and diameter of the golden ear and the shooting distance and pixels, specifically:

[0044] The camera distortion correction module uses equally spaced concentric circles to perform cubic spline fitting on the distortion law of the image, and outputs the corresponding coefficient matrix and fitting function to obtain an accurate image after correction.

[0045] The preliminary pixel-to-actual-size conversion module takes the horizontal (vertical) middle section of the mushroom stick as the reference plane, and concentric circles with a fixed center on the set reference plane as the reference object. The pixel coordinate differences between the diameters (heights) of the reference objects are detected multiple times, and the average value is taken. The ratio of the pixel coordinates to the actual coordinates is calculated using the formula.

[0046]

[0047] Among them, D is the actual length of the circle, k is the scale factor, l is the distance between the camera and the circle, and d is the pixel diameter.

[0048] Then, taking the reference plane as the reference, each group is increased by 0.5 unit length from bottom to top. According to the principle of similar triangles captured by the camera, the above ratio formula is used to calculate the distance and pixel diameter data of all groups, and the predicted diameter (height) value is obtained. The absolute error is obtained by subtracting it from the actual diameter (height), and the model is corrected. The basic ratio of the calculation is obtained through this linear fitting method. The subsequent use of this ratio model to adjust the pixel ratio deviation caused by the distance change is called the pixel ratio adjustment method.

[0049] The corrected image in step S3 is input into the golden ear positioning detection model, and a pixel-level mask and an initial growth cycle classification are output;

[0050] like Figure 2As shown, the golden ear positioning detection model includes a feature extraction backbone network, a feature fusion network and a detection head; the golden ear positioning detection model is designed based on a multi-task loss function.

[0051] The process of the feature extraction network is specifically as follows:

[0052] Video stream images are acquired in real time through two remote cameras. First, necessary preprocessing operations are performed on the images, including size scaling and normalization, so that the golden ear images meet the requirements of model input.

[0053] The preprocessed image is input into the feature extraction backbone network. In this stage, the image is passed through a series of convolutional layers to extract multi-scale features.

[0054] And further enhance the feature expression through the feature fusion network. The specific process is as follows:

[0055] The image is first passed through multiple convolutional layers to extract shallow features to retain the detailed information in the image;

[0056] Then, deep features are extracted through layer-by-layer convolution to extract high-level semantic information of the image;

[0057] The extracted high-level semantic information is input into the feature fusion network to achieve feature enhancement.

[0058] The feature-enhanced image is passed through the spatial pyramid pooling module to further enhance the receptive field and integrate features of different scales, so that the model can better handle objects of different sizes and output multi-layer feature maps.

[0059] The process of the feature fusion network, in which the multi-layer feature maps output by the feature extraction network are further fused through multiple upsampling, downsampling and splicing operations to obtain a feature map containing multi-scale information, is specifically as follows:

[0060] The low-resolution feature map is upsampled and fused with the high-resolution feature map to retain more spatial detail information;

[0061] Feature maps are spliced ​​at different resolutions to enrich the expression dimension of features, thereby enhancing the model's ability to detect objects of different scales;

[0062] After being processed by the multi-scale fusion module, the output feature map has stronger expressive power and is passed to the detection head stage.

[0063] The main task of the detection head is to detect, locate, classify and generate segmentation masks for the targets in the image. The specific steps are as follows:

[0064] Through the mask coefficient module, a set of mask coefficients are generated, which are used to define the segmentation areas of different objects in the image;

[0065] Through the detection module, each target is classified and bounding box predicted to determine the target category and location information;

[0066] Generate prototype masks, which represent the basic contour information of the target and are combined with the coefficients generated by the mask coefficient module to generate the final segmentation mask;

[0067] Through the non-maximum suppression module, overlapping bounding boxes are removed and only the most representative detection results are retained, ensuring that only one detection box is output for each target.

[0068] The golden ear positioning detection model uses a combined loss function to simultaneously achieve target detection and semantic segmentation tasks. The combined loss function consists of three parts, specifically:

[0069] The bounding box overlap loss is used to calculate the geometric error between the bounding box predicted by the model and the true bounding box. This loss measures the accuracy of the model in predicting the size and position of the bounding box. CIoU loss is used to calculate this error. CIoU loss can comprehensively consider the overlapping area, center position and aspect ratio of the bounding box, thereby effectively improving the positioning accuracy of the target.

[0070] The bounding box localization loss is used to improve the bounding box regression accuracy in object detection. By modeling the fine-grained discrete distribution of bounding box coordinates, the model is more flexible and accurate in locating bounding boxes.

[0071] The segmentation loss is used to quantify the difference between the segmentation mask predicted by the model and the true segmentation mask, and the binary cross entropy loss is used to calculate the error of the mask.

[0072] The system renders the output detection and segmentation results visually, displays the location of each golden ear and its growth stage classification on the image, and uses the generated segmentation mask to cover the target area on the image, so that users can clearly see the outline of the target and the area it occupies.

[0073] This model adopts the model training strategy of transfer learning: first pre-training on a large-scale dataset, and then fine-tuning on a small number of samples collected in the actual application environment;

[0074] Pre-training is performed on a selected large-scale dataset to initialize the model weights, so that the model can learn rich feature representations and improve its generalization ability in subsequent tasks;

[0075] Perform data enhancement on a small number of samples collected in the actual application environment, such as rotation, scaling, and cropping, to increase data diversity and improve the model's adaptability to golden ear images in different states;

[0076] The pre-trained model is fine-tuned on the golden ear image samples after data enhancement, and the model parameters are adjusted so that the model can accurately locate and segment the boundaries and shapes of the golden ears based on accurate classification.

[0077] The corrected image and the output results of the golden ear positioning detection model are input into the golden ear height and diameter calculation model to obtain the height, diameter, positioning and shape of the golden ear, which are then compared with the golden ear growth feature database to obtain the growth cycle of the golden ear.

[0078] The positioning refers to obtaining the position of the golden ear in the image through dual cameras respectively, and combining them to construct the position (x, y, z) of the golden ear in three-dimensional space.

[0079] The gold ear height and diameter calculation model is as follows: Figure 4 As shown, the method uses computer vision technology and multi-parameter linear fitting technology to measure the height and diameter of the important indicators that can reflect the growth of golden ear. It includes the following steps:

[0080] Establishing a database of growth characteristics of golden ear by collecting and analyzing multi-period, large-scale growth-related data of golden ear through statistical and regression methods;

[0081] Based on the corrected image, the height and diameter characteristics of the golden ear are estimated in the image space, and then the height and diameter of the cap are calculated more accurately through the pixel ratio deviation, so as to obtain the depth information of the golden ear from any camera, obtain the coordinates of each key point in the golden ear outline in three-dimensional space and the morphological parameters of the cap, and obtain the tangent vector and normal vector at its center of mass, so as to obtain the actual height and diameter of the golden ear cap.

[0082] Then, the actual height and diameter obtained are compared with the output information obtained by the golden ear positioning detection model using the golden ear growth characteristic database to output the final growth cycle judgment.

[0083] The pixel ratio deviation is further obtained by the ratio of the image pixel coordinates obtained by the proportional relationship model to the actual coordinates.

[0084] The Tremella fuciformis growth characteristic database used in this embodiment is shown in Table 1:

[0085] Table 1

[0086]

[0087] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A vision-based method for intelligent positioning and maturity estimation of golden ear, characterized in that: include: Get real-time images of golden ear; Input the real-time image into the proportional relationship model to obtain the corrected image; The rectified image is input into the golden ear location detection model, which outputs the pixel-level mask and the initial growth cycle classification; The corrected image and the output results of the golden ear positioning detection model are input into the golden ear height and diameter calculation model to obtain the height, diameter, positioning and morphology of the golden ear, which are then compared with the golden ear growth feature database to obtain the growth cycle of the golden ear.

2. The method for intelligent positioning and maturity estimation of golden ear according to claim 1, characterized in that: The real-time image of the golden ear is taken by dual cameras, which are perpendicular to each other and respectively take a top view and a side view of the golden ear growing on the mushroom stick.

3. The method for intelligent positioning and maturity estimation of golden ear according to claim 1, characterized in that: The proportional relationship model is used to correct camera distortion and achieve preliminary conversion from pixels to actual size, specifically: Using equally spaced concentric circles, cubic spline fitting is performed on the barrel distortion law of the image, and the corresponding coefficient matrix and fitting function are output; Correct the image according to the output coefficient matrix and fitting function; The central cross section or vertical section of the mushroom stick is determined as the reference plane, the pixel coordinate difference between the reference plane and the diameter or height of the reference object is detected multiple times, the ratio of the pixel coordinate to the actual coordinate is calculated, the pixel coordinate difference between the diameter or height of the reference object is detected multiple times, and the average value is taken to obtain the ratio of the image pixel coordinate to the actual coordinate.

4. The method for intelligent positioning and maturity estimation of golden ear according to claim 1, characterized in that: The golden ear positioning detection model includes a feature extraction backbone network, a feature fusion network and a detection head; Feature extraction backbone network: used to extract multi-layer feature maps of the rectified image, including multi-layer convolutional layers, layer-by-layer deep convolution, feature fusion and spatial pyramid; Feature fusion network: used to obtain feature maps of multi-scale information of multi-layer feature maps, including up-sampling and down-sampling modules, feature fusion modules, splicing modules and multi-scale fusion modules; Detection head: used to obtain pixel-level masks and initial maturity classification.

5. The method for intelligent positioning and maturity estimation of golden ear according to claim 4, characterized in that: The golden ear positioning detection model adopts a combined loss function, which combines the bounding box overlap loss, the bounding box positioning loss and the segmentation loss.

6. The method for intelligent positioning and maturity estimation of golden ear according to claim 5, characterized in that: The golden ear positioning detection model adopts a model training strategy of transfer learning.

7. The method for intelligent positioning and maturity estimation of golden ear according to claim 1, characterized in that: The golden ear height and diameter calculation model specifically obtains the height and diameter features of the golden ear through the corrected image, and then adjusts the height and diameter of the cap through the pixel ratio deviation, thereby obtaining the depth information of the golden ear from any camera, obtaining the coordinates of each key point in the golden ear contour in three-dimensional space and the morphological parameters of the cap, and obtaining the tangent vector and normal vector at its center of mass, thereby obtaining the actual height and diameter of the golden ear cap.

8. The method for intelligent positioning and maturity estimation of golden ear according to any one of claims 1 to 7, characterized in that: The growth cycle of golden ear includes primordium stage, opening stage, differentiation stage, color change stage and maturity stage.

9. The method for intelligent positioning and maturity estimation of golden ear according to claim 8, characterized in that: A visualization step is also included.

10. The method for intelligent positioning and maturity estimation of golden ear according to claim 1, characterized in that: The golden ear growth characteristic database is constructed based on the golden ear growth data, and the golden ear growth data includes the reference range of the golden ear diameter, height and color corresponding to the growth cycle of the golden ear.

Images