Method for obtaining growing information of living plants in a plant factory based on oblique image

Abstract

The application discloses a method for acquiring growing information of lettuce in a plant factory based on oblique shooting images, which adopts a camera to shoot the lettuce in the plant factory at a specific oblique angle, and then performs pretreatment on the obtained lettuce image, and the processing procedure comprises the following steps: data labeling, mask-rcnn instance segmentation model training, data screening, construction of a single lettuce data set, image processing operation is performed on the obtained image, partial phenotypic characteristics (inclined projection area, color information and texture complexity) of the lettuce are acquired, and a fresh weight estimation model is trained by using a convolutional neural network, and growing detection of the lettuce is realized. The application solves the problem that it is difficult to acquire growing information of crops in a closed and non-open space such as a plant factory planting rack, and has the characteristics of high accuracy and high efficiency, and is helpful to promote the construction of unmanned plant factories and to empower intelligent agriculture.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture, specifically to a method for acquiring lettuce growth information in a plant factory based on oblique images. Background Technology

[0002] In existing technologies, the non-destructive acquisition of crop growth information using digital images often employs side or top views. Researchers typically capture the crop's frontal or top-front views to obtain information about its height and canopy projection area. However, plant factories, as typical examples of vertical agriculture, utilize three-dimensional planting racks for crop cultivation and implement strict control over the growth environment. To achieve higher yields per unit area and more efficient energy conversion, the spacing between the planting rack layers is usually quite close, meaning the crop's growth area is very limited. Vertically, it's difficult to find extra space to install an imaging system. Furthermore, plant factories use artificial light strips, and the acquisition equipment, mounted at the top, may block the light needed for crop growth, affecting its normal development. Placing the acquisition equipment on the side, due to the plant density, results in severe occlusion of the crop in the captured images, with the side-front view only able to monitor the outermost crops. In conclusion, due to the structural limitations of plant factories, conventional methods of capturing frontal, side, and top-front views of the plants are insufficient to obtain crop growth information. Summary of the Invention

[0003] This invention overcomes the shortcomings of existing technologies by providing a method for acquiring lettuce growth information in a plant factory based on oblique images. The method uses oblique imaging, which solves the problem of difficulty in acquiring crop growth information in plant factories. Furthermore, it uses neural networks to correct the distortion problem caused by oblique imaging, resulting in high accuracy and efficiency. This helps to promote the construction of unmanned plant factories and empowers smart agriculture.

[0004] The technical solution of this invention to solve the problems of the prior art is:

[0005] A method for obtaining lettuce growth information in a plant factory based on oblique images includes the following steps:

[0006] S1: Use a camera to capture images of lettuce growth inside the plant factory by taking angled shots;

[0007] S2: Input the acquired growth image into the instance segmentation model to segment out individual lettuce images;

[0008] S3: Recognize the segmented individual lettuce images and identify the first four rows of lettuce images;

[0009] S4: Send the identified lettuce image into the feature extraction model to obtain the inclined projection area, perimeter, color information and texture features through feature extraction;

[0010] S5: Send the identified lettuce image into the fresh weight estimation model to estimate the fresh weight value of the lettuce;

[0011] S6: Output the growth information of the corresponding lettuce, which includes the lettuce characteristics and the fresh weight value of the lettuce.

[0012] Preferably, in step S1, the step of acquiring the lettuce growth image in the plant factory is:

[0013] S11: According to the type and scale of the lettuce to be detected, adjust the installation position and oblique shooting angle of the camera, and set up the image acquisition platform;

[0014] S12: Take pictures of the lettuce images in the whole growth period at regular intervals and fixed points, and record the fresh weight value of the photographed lettuce at the same time.

[0015] Preferably, in step S2, the segmentation step of the instance segmentation model is:

[0016] S21: After collecting the oblique shooting images of the lettuce, use LabelMe data labeling software to perform data labeling work on the oblique shooting images, and the labeling type is divided into two categories, which are single lettuce image (1) and background (0); wherein, the oblique shooting image only needs to cover the area to be monitored, and the area containing the lettuce is replaced by pixels, and the pixel value of the area is set to 0;

[0017] S22: Use OpenCV to perform data augmentation on the data set to obtain the labeled data set;

[0018] S23: Divide the obtained labeled data set into training set and test set in the ratio of 8:2;

[0019] S24: Send the divided training set and test set into the data list of the Mask-RCNN instance segmentation model built in the integrated model MMDetection, and perform model training;

[0020] S25: Call the Mask-RCNN instance segmentation model and the trained pth file model parameters to process all collected lettuce images;

[0021] S26: Change the use function of the integrated model MMDetection, use the where function of the numpy function library to extract the data of the area represented by each lettuce in the output tensor list variable; use the array function of the numpy function library to convert the encoded numerical value into octal format;

[0022] S27: The mask image of the original image size and the original image are used to extract the area covered by the same area using the bitwise_or function of OpenCV, and finally the image of a single lettuce plant is obtained. This image retains the position information of the single lettuce plant in the original image.

[0023] Preferably, in step S22, the data augmentation operations include color space conversion, adding noise, horizontal flipping, and changing brightness.

[0024] Preferably, in step S24, the input image size is 1333ppi x 800ppi, the learning rate is set to 0.02, the batch size is set to 32, and SGD is used as the loss function.

[0025] Preferably, in step S3, the step of identifying the first four rows of lettuce images is as follows:

[0026] S31: Use OpenCV's findCounters function to identify lettuce regions;

[0027] S32: Use OpenCV's minAreaRect function to find the minimum bounding rectangle of the lettuce region;

[0028] S33: Calculate the center point of the smallest bounding rectangle as the coordinates (x, y) of the lettuce. The calculation formula is as follows:

[0029] P1 = (x1, y1);

[0030] P2 = (x2, y2);

[0031] P3 = (x3, y3);

[0032] P4 = (x4, y4);

[0033] In the formula, P1, P2, P3, and P4 are the coordinates of the four vertices of the identified minimum bounding matrix, and the center coordinates of a single lettuce plant are:

[0034]

[0035]

[0036] In the formula, x i Let x be the set of x-coordinates of the four corner points of the rectangle, and y be the set of x-coordinates of the four corner points of the rectangle. i The set of y-coordinates of the four corner points of the rectangle;

[0037] S33: Use steps S31-S33 to collect the center coordinates of each lettuce plant in all the individual lettuce images in a single original image;

[0038] S34: using the K value clustering method, the longitudinal coordinate value of the center point of all single lettuce is divided into different categories according to the size of the longitudinal coordinate value, and finally the four groups with the largest y value are taken as the output of the first four rows of lettuce images;

[0039] S35: input the output of the first four rows of lettuce images into the ByteTrack algorithm model to avoid repeated counting of video stream.

[0040] Preferably, in step S4, the feature extraction step of the feature extraction model is:

[0041] S41: according to the lettuce area identified in step S31, the projection area of the lettuce is calculated using the contourArea function of the OpenCV function library;

[0042] S42: according to the lettuce area identified in step S31, the color feature of the lettuce is calculated using the cvtColor function and the count function of the OpenCV function library, and the color feature includes the mean and median of the total pixel value in six image spaces of rgb, hsv, ycrcb, hls, lab and luv, wherein the calculation formula of the mean is as follows:

[0043]

[0044] In the formula, X is the pixel value of the original image single channel; N is the pixel number of the lettuce area;

[0045] S43: according to the lettuce area identified in step S31, the texture feature of the lettuce is calculated using the skimage function library feature.graycoprops function, feature.graycoprops function and feature.graycoprops function;

[0046] S44: according to the lettuce area identified in step S31, the circumference of the lettuce is calculated using the arcLength function of the OpenCV function library.

[0047] Preferably, in step S4, the step of estimating the fresh weight of the lettuce is:

[0048] S51: using the single lettuce image of the target area extracted in step S3, the single lettuce is corresponding to the real fresh weight value, and the data set for fresh weight estimation is constructed;

[0049] S52: the constructed database is subjected to data enhancement operation;

[0050] S53: Put the enhanced data set into the constructed Disnet model for training; the input is an image of 224ppix224ppi, the output is the estimated fresh weight value of the lettuce, the true value is the actually measured fresh weight value of the lettuce, the MSELoss function is used as the loss function, and a fresh weight estimation model is trained;

[0051] S54: The first four rows of lettuce images extracted in step S3 are sequentially sent into the trained fresh weight estimation model to obtain the estimated fresh weight value.

[0052] Preferably, in step S52, the enhancement types of the data enhancement operation are: horizontal flip, noise increase, lightness change and horizontal displacement of the lettuce image in the background.

[0053] Compared with the prior art, the present application has the following beneficial effects:

[0054] 1. The method for obtaining lettuce growth information in a plant factory based on an oblique shooting image of the present application solves the problems of low efficiency of top shooting and easy blocking of lettuce caused by side shooting, and further solves the problem of difficulty in obtaining crop information in a closed and non-open space such as a plant factory by using an oblique shooting angle to obtain the growth information of lettuce in a plant factory.

[0055] 2. The method for obtaining lettuce growth information in a plant factory based on an oblique shooting image of the present application solves the problem of perspective distortion of oblique shooting images by using a convolutional neural network based method to construct a regression model and constructing a Disnet network for oblique shooting, and improves the accuracy of fresh weight estimation and the efficiency of growth information acquisition by training a large amount of data. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 It is a schematic diagram of the laboratory used in the method for obtaining lettuce growth information in a plant factory based on an oblique shooting image of the present application.

[0057] Figure 2 It is a schematic diagram of the image acquisition system.

[0058] Figure 3 It is a schematic diagram of labeling a single lettuce plant in an image.

[0059] Figure 4 It is a horizontally flipped image.

[0060] Figure 5 It is an image processed by changing lightness.

[0061] Figure 6 It is an image after adding noise.

[0062] Figure 7 It is an image after color space conversion.

[0063] Figure 8 Loss function of the instance segmentation model.

[0064] Figure 9 Accuracy curve of the instance segmentation model.

[0065] Figure 10 Original image.

[0066] Figure 11 Single lettuce image extracted from the original image in Figure 10 by the instance segmentation model.

[0067] Figure 12 Effect diagram after target tracking by the ByteTrack algorithm.

[0068] Figure 13 Diagram for embodying the performance of the training set.

[0069] Figure 14 Disnet network architecture diagram.

[0070] Figure 15 Flowchart of the method for obtaining lettuce growth information in a plant factory based on an oblique image according to the present application. DETAILED DESCRIPTION

[0071] The present application will be further described in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present application are not limited thereto.

[0072] Referring to Figures 1-15 , the method for obtaining lettuce growth information in a plant factory based on an oblique image according to the present application comprises the following steps:

[0073] S1: using a camera to obtain growth images of lettuces in a plant factory by an oblique shooting method;

[0074] S2: sending the obtained growth images into an instance segmentation model to segment out single lettuce images;

[0075] S3: identifying the segmented single lettuce images to identify the lettuce images in the first four rows;

[0076] S4: sending the identified lettuce images into a feature extraction model to obtain the inclined projection area, the circumference, the color information and the texture features by a feature extraction method;

[0077] S5: sending the identified lettuce images into a fresh weight estimation model to estimate the fresh weight value of the lettuces;

[0078] S6: outputting the growth information of the corresponding lettuces, wherein the growth information comprises the lettuce features and the fresh weight value of the lettuces.

[0079] In step S1, the step of acquiring the lettuce growth image in the plant factory is:

[0080] S11: According to the type and scale of the lettuce to be detected, adjust the installation position and oblique shooting angle of the camera, and set up the image acquisition platform;

[0081] S12: Time and point shooting of lettuce images in the whole growth period, and recording the fresh weight value of the shooting lettuce.

[0082] In step S2, the instance segmentation step of the instance segmentation model is:

[0083] S21: After collecting the oblique shooting image of the lettuce, use LabelMe data labeling software to perform data labeling work on the oblique shooting image, and the labeling type is divided into two categories, which are single lettuce image (1) and background (0); wherein, the oblique shooting image only needs to cover the area to be monitored, and the area containing the lettuce in the rear row is replaced by pixels, and the pixel value of the area is set to 0;

[0084] S22: Use OpenCV to perform data augmentation on the data set to obtain the labeled data set;

[0085] S23: Divide the obtained labeled data set into training set and test set in the ratio of 8:2;

[0086] S24: Put the divided training set and test set into the data list of the Mask-RCNN instance segmentation model built in the integrated model MMDetection, and train the model;

[0087] S25: Call the Mask-RCNN instance segmentation model and the trained pth file model parameters to process all collected lettuce images;

[0088] S26: Change the use function of the integrated model MMDetection, use the where function of the numpy function library to extract the data of each lettuce represented area in the output tensor list variable; use the array function of the numpy function library to convert the encoded numerical value to octal format;

[0089] S27: Use the bitwise_or function of OpenCV to extract the common covered area of the obtained mask image of the original image size and the original image, and finally obtain the single lettuce image, which retains the position information of the single lettuce image in the original image.

[0090] In step S22, the data augmentation operation includes color space conversion, noise increase, horizontal flip and brightness change.

[0091] In step S24, the input image size is 1333ppix800ppi, the learning rate is set to 0.02, the batchsize is set to 32, and SGD is used as the loss function.

[0092] In step S3, the steps for identifying the first four rows of lettuce images are:

[0093] S31: Identify the lettuce area using the findCounters function of OpenCV;

[0094] S32: Find the minimum circumscribed rectangle of the lettuce area using the minAreaRect function of OpenCV;

[0095] S33: Calculate the center point of the minimum circumscribed rectangle as the coordinates (x, y) of the lettuce, and the calculation formula is as follows

[0096] P1 = (x1, y1);

[0097] P2 = (x2, y2);

[0098] P3 = (x3, y3);

[0099] P4 = (x4, y4);

[0100] where P1, P2, P3, and P4 are the coordinates of the four vertices of the identified minimum circumscribed rectangle, and the center coordinates of the single lettuce are:

[0101]

[0102]

[0103] where x i is a set of x-coordinate values of the four corners of the rectangle, and y i is a set of y-coordinate values of the four corners of the rectangle.

[0104] S33: Use steps S31-S33 on all single lettuce images in the original image to collect the center coordinates of each single lettuce;

[0105] S34: Use K-value clustering method to divide the vertical coordinate values of all single lettuce center points into different categories according to the size of the vertical coordinate values, and finally take the 4 groups with the largest y values as the output of the first four rows of lettuce images.

[0106] S35: Input the output of the first four rows of lettuce images into the ByteTrack algorithm model to perform multi-target tracking to avoid repeated counting of video streams.

[0107] In step S4, the feature extraction step of the feature extraction model is:

[0108] S41: According to the lettuce area identified in step S31, the projection area of the lettuce is calculated using the contourArea function of the OpenCV function library;

[0109] S42: According to the lettuce area identified in step S31, the color features of the lettuce are calculated using the cvtColor function and the count function of the OpenCV function library, which include the mean and median of the total pixel values in six image spaces of rgb, hsv, ycrcb, hls, lab and luv. The calculation formula of the mean is as follows:

[0110]

[0111] In the formula, X is the pixel value of the original image single channel; N is the pixel number of the lettuce area;

[0112] S43: According to the lettuce area identified in step S31, the texture features of the lettuce are calculated using the skimage function library feature.graycoprops function, feature.graycoprops function and feature.graycoprops function;

[0113] S44: According to the lettuce area identified in step S31, the circumference of the lettuce is calculated using the arcLength function of the OpenCV function library.

[0114] In step S4, the step of estimating the fresh weight of the lettuce is:

[0115] S51: Using the single lettuce image of the target area extracted in step S3, the single lettuce is corresponded to its true fresh weight value, and a data set for fresh weight estimation is constructed;

[0116] S52: The constructed database is subjected to data enhancement operation;

[0117] S53: The enhanced data set is put into the constructed Disnet model for training; the input is 224ppix224ppi image, the output is the estimated fresh weight value of the lettuce, the true value is the actual measured fresh weight value of the lettuce, the MSELoss function is used as the loss function, and the fresh weight estimation model is trained;

[0118] S54: The first four rows of lettuce images extracted in step S3 are sequentially sent into the trained fresh weight estimation model to obtain the estimated fresh weight value.

[0119] In step S52, the enhancement type of the data enhancement operation is: horizontal flip, increase noise, change in brightness and horizontal displacement of the lettuce image in the background.

[0120] Specific implementation cases:

[0121] The method for obtaining lettuce growth information in a plant factory based on oblique shooting images according to the application comprises the following steps:

[0122] Step one, obtaining the growth image of lettuce in the plant factory;

[0123] In July 2022, an image acquisition platform (see Figure 1 ) was built in a small plant factory constructed by South China Agricultural University (see Figure 2 ), and growth images of red butter lettuce were shot. Among them, 4 planting trays were placed in the planting rack of the plant factory, and the planting trays were arranged in parallel and had consistent height, and each planting tray used 4x8 planting holes for leaf planting. The camera platform was fixed on the side of the planting area using aluminum profiles, and the platform was erected at a position 36 cm away from the top of the planting tray and 36 cm away from the planting rack, and the camera was vertically facing the planting area to shoot at a 21° angle. The camera used S50-IMX291 type camera, and the pixel of the shot image was 3840x2160, the pixel of the video picture was 1080x720, and the frame number was 30 frames. From the time when the seedlings could be seen with the naked eye growing out of the planting hole, about 4 days after transplanting, shooting was carried out every morning at 10-11 o'clock, and shooting was stopped after the leafy vegetables grew a lot. 14 points were selected for fixed-point photography, and three sections of video were recorded. Fresh weight detection of single leafy vegetable was carried out at the same time of shooting images. The weighing instrument used JCS-3103C type precision electronic scale of Diheng Company, with an accuracy of 0.001. The weighing experiment was carried out immediately after shooting the image, and finally the fresh weight information and growth position of the lettuce were recorded according to the shooting points; finally, the image information and fresh weight information of the leafy vegetables from 6 to 21 days were obtained, including 224 pictures and 48 videos.

[0124] Step two, training an instance segmentation model to segment out single lettuce image;

[0125] 1. The original image is labeled using LabelMe software, and the last four rows of lettuce images are covered with a black mask, finally 220 labeled images are obtained, containing 4473 lettuce marking boxes, and the label type is as shown in Figure 3 ;

[0126] 2. The image is data enhanced using color space conversion, noise increase, horizontal flip, and brightness change, finally 2420 images and corresponding json label files are obtained; among them, the enhanced data set type is shown inFigures 4-7 Fig. 1 shows the image after horizontal flip processing, wherein, Figure 4 Fig. 2 shows the image after changing brightness processing; Figure 5 Fig. 3 shows the image after adding noise processing; Figure 6 Fig. 4 shows the image after color space conversion processing. Figure 7

[0127] 3. The final data set is divided into training set and test set in the ratio of 8:2, and is sent to the instance segmentation model for training, wherein the image size of the input model is 1333x800, the learning rate is set to 0.02, the batchsize is set to 32, the SGD is used as the loss function, the epoch of training is set to 24, the loss curve in the final model training result is as shown in Fig. 5, and the accuracy curve is as shown in Fig. 6; the instance segmentation model training result is as follows: Figure 8 Figure 9

[0128] Evaluation index Mmap ]]> ​ mAP@0.5 mAP@0.75 Numerical value 0.821 0.979 0.937

[0129] In the above table:

[0130] Mmap: mean average precision;

[0131] map@0.5: mean average precision value under the Iou threshold of 0.5;

[0132] map@0.75: mean average precision value under the Iou threshold of 0.75;

[0133] 4. The trained instance segmentation model is used to extract the image of single lettuce, which is saved in jpeg format (see Fig. 7). Figures 10-11

[0134] Step three, identify the first four rows of lettuce images;

[0135] 1. Calculate the position coordinates of the extracted lettuce image in the original image;

[0136] 2. Send the y coordinate value in the coordinates into the K value clustering algorithm of the python function library, and return the lettuce image corresponding to the coordinates of the 4 largest y values after classification;

[0137] 3. The pixel box information corresponding to the returned lettuce image is sent into the ByteTrack algorithm model for target tracking, wherein the effect is as shown in Fig. 8. Figure 12

[0138] Step four, extract the phenotype characteristics;

[0139] ​​​​​Using the built-in calculation functions of the OpenCV function library, the projection area, perimeter, color features, and texture features of 1024 lettuce plants were finally obtained, a total of 40 phenotypic characteristics, and part of the data is as follows:

[0140]

[0141] Step five, fresh weight estimation

[0142] 1. Correspond the extracted single red cream lettuce image with its corresponding true fresh weight value, and build a data set for training fresh weight estimation model. The data set includes the rgb image of red cream lettuce and the excel table recording the true fresh weight value of the image. The image storage location is used as the index. Part of the txt data set is as follows:

[0143]

[0144] 2. Perform data augmentation operation on the constructed data set. The main enhancement types are: horizontal flip, noise addition, brightness change and horizontal displacement of lettuce image in the background. The data set increases from 1024 images to 5012 images.

[0145] 3. Put the data set into the constructed Disnet model for training. The input is the image before and after enhancement, the size is 224x224, and the output is the estimated fresh weight. Here, the built-in VGG16 model under the pytorch framework is used for training. The learning rate is quenched. Before 150 epochs, the learning rate is 0.001, and after that, the learning rate is 0.00001. A total of 500 epochs are performed. Batchsize is set to 12, and MSELoss function is used as the loss function. The training result is as shown in Figure 13 , wherein, Figure 13 is the performance of the test set: the determination coefficient on the training set is 0.9730, and the determination coefficient on the test set is 0.9223.

[0146] 4. Put the image of step four to extract the phenotypic characteristics into the trained Disnet model for fresh weight estimation, and output the estimated fresh weight and the extracted phenotypic characteristics at the same time as the monitoring result.

[0147]

[0148] In this embodiment, the Disnet network is constructed, and the specific steps are as follows:

[0149] S1: carry out theoretical analysis; need to add a module capable of extracting position information in the neural network, and also need to extract a module of shallow information, finally fuse the data of shallow information and position information, use the full connection layer to output; wherein the position information extraction module selects Resnet18 as the backbone network. The output of 32 times down-sampling of Resnet structure is subjected to twice average pooling operation, deep semantic information is obtained, and the semantic information is fused with the output of 32 times down-sampling and the output of 16 times down-sampling, to jointly constitute the position information extraction module; the shallow information extraction module mainly consists of three convolutional layers. Through simple convolution operation, the color, texture, shape and other feature information of the lettuce image are extracted, and the spatial information of the image is maximized;

[0150] S2: fuse the output of the position information extraction module and the output of the shallow information extraction module, use twice maximum pooling and twice convolution operation to learn the extracted features, and reduce the dimension of the data. Finally, the fresh weight value of the estimated oblique image is output through three full connection layers; wherein the overall network architecture is as shown in Figure 14

[0151] In order to evaluate the effect of the model, other mature deep learning networks are trained and compared, and the accuracy of other networks is compared with that of the Disnet network. The results are shown in the following table:

[0152]

[0153] By selecting VGG, Resnet and Densenet as the comparison of Disnet network in this embodiment. Because the structure of the feature extraction network in the Disnet network structure is similar to VGG, and the structure of the position information extraction is the output of the intermediate layer of Resnet. In addition, Densenet is a densely connected convolutional neural network, which directly connects all the feature maps of the previous layers to the subsequent layers in a densely connected manner. This densely connected manner can enhance the transmission and reuse of features, improve the training efficiency and accuracy of the network. This feature transmission structure is similar to the position information extraction module of the Disnet structure. Therefore, the above three relatively mature neural network models are used as the comparison of the Disnet model.

[0154] From the results, the data of Densenet is the best, but its performance on the training set is high, reaching 0.99, but the test set is low, which is the phenomenon of overfitting. Compared with other models, the Disnet network model in this embodiment performs well in training accuracy and test accuracy, which confirms the rationality and efficiency of the constructed model.

[0155] ​Data analysis was performed for fresh weight estimation. After feature extraction on the segmented greenhouse lettuce images, the single-plant leaf lettuce image, the true fresh weight of the leaf lettuce, and the extracted feature information were corresponded, and the correlation analysis method (here, the Pearson correlation coefficient analysis) of orange was used to obtain the correlation of each variable feature and fresh weight. The results showed that the correlation coefficient of fresh weight information and inclined projection area was 0.57, and the correlation coefficient of fresh weight information and circumference was 0.53. In addition, the fresh weight information and texture features had high correlation. Further research found that the correlation analysis between the feature information of the single-row lettuce image and the fresh weight value was established, and the correlation coefficient reached 0.98. Taking the data of the fourth row as an example, the correlation coefficient between the fresh weight value of the fourth row image and the extracted inclined projection area was 0.96, the correlation coefficient between the fresh weight value and the circumference was 0.86, and the texture features were all above 0.9. This also indirectly proves the correctness of the phenotype analysis, and the fresh weight of the leaf lettuce is indeed related to the surface area, color information and texture features. These features are good at learning by convolutional neural network, so the regression model based on convolutional neural network has high feasibility for estimating the fresh weight of leaf lettuce.

[0156] Theoretically, the leaf lettuce needs to be observed after 5 days of transplanting to the growth area (the leaf can grow out of the hole). The whole growth and development of the leaf lettuce has a significant change in appearance, from 3-4 leaves at the beginning to 10-20 leaves, and the leaf area also changes significantly. In the image, the leaf lettuce image area becomes larger, and the image complexity also increases significantly due to the increase in the number of leaves. For specific image data, it may also be reflected in the larger leaf shadow area and darker leaf color. This means that deep information features of the image need to be collected to represent the image complexity, shadow feature and other texture feature information. Moreover, due to the use of side shooting, perspective distortion is inevitable, which will cause the image to be larger near and smaller far, and there will be some deformation. However, this change is weak, because the first four rows of images are selected for prediction, and the straight-line distance is less than 2 meters. At the same time, this distortion has a certain rule and conforms to the perspective theorem, which belongs to bilinear change. This change can be learned by deep learning. Therefore, the position information of the image needs to be preserved, which needs to be preserved in the preparation of the data set. When designing the network, the shallow information needs to be considered for fusion.

[0157] The above is the preferred embodiment of the present application, but the embodiments of the present application are not limited by the above, any change, modification, substitution, combination, simplification made without departing from the spirit and principles of the present application, should be an equivalent replacement method, all included in the protection scope of the present application.

Claims

1. A method for acquiring lettuce growth information in a plant factory based on oblique images, characterized in that, Includes the following steps: S1: Use a camera to capture images of lettuce growth inside the plant factory by taking angled shots; S2: Input the acquired growth image into the instance segmentation model to segment out individual lettuce images; S3: Recognize the segmented individual lettuce images and identify the first four rows of lettuce images; S4: The identified lettuce image is fed into the feature extraction model to obtain the tilted projected area, perimeter, color information and texture features through feature extraction; S5: Input the identified lettuce image into the fresh weight estimation model to estimate the fresh weight of the lettuce; the steps for estimating the fresh weight of the lettuce are as follows: S51: Using the single lettuce images of the target area extracted in step S3, the single lettuce is matched with its true fresh weight value to construct a dataset for fresh weight estimation. S52: Perform data augmentation operations on the constructed database; S53: The enhanced dataset is fed into the constructed Disnet model for training; the input is an image of 224ppi, the output is the estimated fresh weight of lettuce, and the true value is the actual measured fresh weight of lettuce. The MSELoss function is used as the loss function to train the fresh weight estimation model; where... The steps to construct a Disnet network are as follows: S531: A location information extraction module and a shallow information extraction module are added to the neural network model. The extracted shallow information and location information are fused and output using a fully connected layer. The location information extraction module uses ResNet 18 as the backbone network. The output of the ResNet structure, downsampled by 32 times, undergoes two average pooling operations to obtain deep semantic information. This deep semantic information is then fused with the output downsampled by 32 times and the output downsampled by 16 times to form the location information extraction module. The shallow information extraction module consists of three convolutional layers used to extract feature information from the lettuce image while preserving the spatial information of the image. The feature information of the lettuce image includes its color, texture, and shape. S532: The outputs of the location information extraction module and the shallow information extraction module are fused, and the extracted features are learned using two max pooling operations and two convolution operations, and the data is reduced in dimensionality; then, the estimated freshness value of the oblique image is output through three fully connected layers. S54: Input the first four rows of lettuce images extracted in step S3 into the trained fresh weight estimation model in sequence to obtain the estimated fresh weight value. S6: Output the growth information of the corresponding lettuce, which includes lettuce characteristics and lettuce fresh weight value.

2. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 1, characterized in that, In step S1, the step of acquiring images of lettuce growth within the plant factory is as follows: S11: Adjust the camera's installation position and angle of view according to the type and scale of lettuce to be inspected, and set up the image acquisition platform; S12: Take images of lettuce throughout its entire growth cycle at fixed times and locations, and record the fresh weight of the lettuce being photographed.

3. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 2, characterized in that, In step S2, the segmentation steps of the instance segmentation model are as follows: S21: After collecting oblique images of lettuce, use LabelMe data annotation software to perform data annotation on the oblique images. The annotation types are divided into two categories: single lettuce images and background. The annotated oblique images only need to cover the area to be monitored. The area containing lettuce in the back row is replaced with pixels, and the pixel value of that area is set to 0. S22: Use OpenCV to augment the dataset to obtain a labeled dataset; S23: Divide the obtained labeled dataset into training and test sets in an 8:2 ratio; S24: Input the divided training and test sets into the data list of the Mask-RCNN instance segmentation model built into the MMDetection ensemble model, and train the model. S25: Use the Mask-RCNN instance segmentation model and the trained pth file model parameters to process all collected lettuce images; S26: Modify the function used by the ensemble model MMDetection to extract the area data represented by each lettuce using the where function of the NumPy library from the output tensor list variable; use the array function of the NumPy library to convert the encoded numerical values ​​into octal format; S27: The mask image of the original image size and the original image are used to extract the area covered by the same area using the bitwise_or function of OpenCV, and finally the image of a single lettuce plant is obtained. This image retains the position information of the single lettuce plant in the original image.

4. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 3, characterized in that, In step S22, the data augmentation operations include color space conversion, adding noise, horizontal flipping, and changing brightness.

5. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 3, characterized in that, In step S24, the input image size is 1333ppi x 800ppi, the learning rate is set to 0.02, the batch size is set to 32, and SGD is used as the loss function.

6. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 3, characterized in that, In step S3, the steps for identifying the first four rows of lettuce images are as follows: S31: Use OpenCV's findCounters function to identify lettuce regions; S32: Use OpenCV's minAreaRect function to find the minimum bounding rectangle of the lettuce region; S33: Calculate the center point of the smallest bounding rectangle as the coordinates (x, y) of the lettuce. The calculation formula is as follows: ； ； ； ； In the formula, P1, P2, P3, and P4 are the coordinates of the four vertices of the identified minimum bounding matrix, and the center coordinates of a single lettuce plant are: ； ； In the formula, The set of x-coordinates of the four corner points of the rectangle. The set of y-coordinates of the four corner points of the rectangle; S33: Use steps S31-S33 to collect the center coordinates of each lettuce plant in all the individual lettuce images in a single original image; S34: Using the K-value clustering method, the y-coordinate values ​​of the center points of all individual lettuce plants are divided into different categories according to the magnitude of the y-coordinate values. Finally, the four groups with the largest y-values ​​are selected as the first four rows of lettuce images for output. S35: Input the first four rows of lettuce images into the ByteTrack algorithm model for multi-target tracking to avoid duplicate counting in the video stream.

7. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 6, characterized in that, In step S4, the feature extraction steps of the feature extraction model are as follows: S41: Based on the lettuce region identified in step S31, calculate the projected area of ​​the lettuce using the contourArea function from the OpenCV library; S42: Based on the lettuce region identified in step S31, the cvtColor and count functions of the OpenCV function library are used to calculate the color features of the lettuce. The color features include the mean and median of the total pixel values ​​in six image spaces: RGB, HSV, YCRCB, HLS, LAB, and LUV. The formula for calculating the mean is as follows: ； In the formula: X is the pixel value of a single channel of the original image; N is the number of pixels in the lettuce region; S43: Based on the lettuce region identified in step S31, use the feature.graycoprops function, feature.graycoprops function, and feature.graycoprops function from the skimage library to calculate the texture features of the lettuce; S44: Based on the lettuce region identified in step S31, calculate the perimeter of the lettuce using the arcLength function of the OpenCV function library.

8. The method for acquiring lettuce growth information in a plant factory based on oblique images according to claim 1, characterized in that, In step S52, the data augmentation operation is of the following types: horizontal flipping, adding noise, changing brightness, and horizontal displacement of the lettuce image in the background.

Images