Leaf vegetable fresh weight estimation method, device and equipment

Abstract

The invention provides a leaf vegetable fresh weight estimation method, device and equipment, and relates to the technical field of leaf vegetable fresh weight estimation. The method comprises the following steps: inputting a leaf vegetable image, a visible light vegetation index, geometric information and color information into a leaf vegetable fresh weight prediction model, and outputting a leaf vegetable fresh weight prediction result; the leaf vegetable fresh weight prediction model is used for estimating the fresh weight of the leaf vegetables; the leaf vegetable fresh weight prediction model comprises an initial feature extraction module, a multi-scale feature extraction module and a multi-dimensional feature fusion module. The method provided by the embodiment of the invention can effectively improve the accuracy of the fresh weight estimation result of the leaf vegetables.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of leafy vegetable fresh weight estimation, and in particular to a leafy vegetable fresh weight estimation method, device and equipment. BACKGROUND

[0002] In vegetable production, early prediction of vegetable yield is of great significance for planting planning and market arrangement. The yield of leafy vegetables is equivalent to their fresh weight. Therefore, estimating the fresh weight of leafy vegetables is very important for agricultural production.

[0003] Traditional crop fresh weight determination methods are often destructive and rely on manual measurement, with low data collection efficiency, large errors, high labor costs, and inability to continuously and dynamically obtain vegetable growth parameters. SUMMARY

[0004] The present application provides a leafy vegetable fresh weight estimation method, device and equipment, which can effectively improve the accuracy of leafy vegetable fresh weight estimation results and realize end-to-end leafy vegetable fresh weight estimation. The present application integrates vegetation index into the fresh weight estimation network model, effectively fusing multiple features with only a small number of parameters and computing costs. Experiments show that the network has the advantages of low computing cost, light weight and high precision.

[0005] The present application provides a leafy vegetable fresh weight estimation method, comprising the following steps.

[0006] Obtaining a leafy vegetable image of a first target type; According to the leafy vegetable image, extracting the visible light vegetation index, geometric information and color information; Inputting the leafy vegetable image, visible light vegetation index, geometric information and color information into a leafy vegetable fresh weight prediction model to output a prediction result of the leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model is used for estimation of the leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model comprises an initial feature extraction module, a multi-scale feature extraction module and a multi-dimensional feature fusion module.

[0007] The present application also provides a leafy vegetable fresh weight estimation device, comprising the following modules: The obtaining module is configured to obtain a leafy vegetable image of a first target type; The determining module is configured to extract the visible light vegetation index, geometric information and color information according to the leafy vegetable image; An estimation module is configured to input the leaf vegetable image, the visible light vegetation index, the geometric information and the color information into a leaf vegetable fresh weight prediction model, and output a prediction result of the leaf vegetable fresh weight; the leaf vegetable fresh weight prediction model is used for estimation of the leaf vegetable fresh weight; the leaf vegetable fresh weight prediction model comprises an initial feature extraction module, a multi-scale feature extraction module and a multi-dimensional feature fusion module; The initial feature extraction module is configured to extract initial leaf vegetable image features, initial visible light vegetation index features, initial geometric features and color features from the leaf vegetable image, the visible light vegetation index, the geometric information and the color information. The multi-scale feature extraction module is configured to obtain multi-scale features corresponding to the leaf vegetable image. The multi-dimensional feature fusion module is configured to fuse the initial leaf vegetable image features, the initial visible light vegetation index features, the initial geometric features and color features, and the multi-scale features corresponding to the leaf vegetable image, to obtain a multi-dimensional feature fusion result, and output an estimation result of the leaf vegetable fresh weight.

[0008] The present application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the leaf vegetable fresh weight estimation method according to any one of the above when executing the program.

[0009] The present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the leaf vegetable fresh weight estimation method according to any one of the above.

[0010] The present application also provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the leaf vegetable fresh weight estimation method according to any one of the above.

[0011] The leaf vegetable fresh weight estimation method, device and equipment provided by the present application can comprehensively and accurately estimate the leaf vegetable fresh weight based on multiple vegetation information such as the coverage, growth vigor and health status, chlorophyll content and biomass of the leaf vegetable, thereby improving the accuracy of the leaf vegetable fresh weight estimation result. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to make the technical solutions in the present application or prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on the accompanying drawings also belong to the protection scope of the present application.

[0013] Figure 1 is one of the flowcharts of the leafy vegetable fresh weight estimation method provided by the present application.

[0014] Figure 2 is another flowchart of the leafy vegetable fresh weight estimation method provided by the present application.

[0015] Figure 3 is a schematic diagram of feature extraction provided by the present application.

[0016] Figure 4 is a schematic diagram of multi-scale feature extraction provided by the present application.

[0017] Figure 5 is a schematic diagram of multi-scale feature fusion provided by the present application.

[0018] Figure 6 is a third flowchart of the leafy vegetable fresh weight estimation method provided by the present application.

[0019] Figure 7 is a fourth flowchart of the leafy vegetable fresh weight estimation method provided by the present application.

[0020] Figure 8 is a structural schematic diagram of the leafy vegetable fresh weight estimation device provided by the present application.

[0021] Figure 9 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0022] In order to make the technical solutions in the present application or prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on the accompanying drawings also belong to the protection scope of the present application.

[0023] The leafy vegetable fresh weight estimation method, device and equipment of the present application will be described below. Figures 1-9

[0024] ​To facilitate a clearer understanding of the technical solutions of the various embodiments of this application, some technical content related to the various embodiments of this application will be introduced first.

[0025] Fresh weight estimation of leafy vegetables is crucial for efficient cultivation and variety breeding, providing vital data support for intelligent production decisions in plant factories. However, traditional methods for measuring crop fresh weight are often destructive, rely on manual measurement, resulting in low data acquisition efficiency, large errors, high labor costs, and an inability to continuously and dynamically acquire vegetable growth parameters. The development of computer vision technology has provided a new method for in-situ, non-destructive fresh weight detection based on images, effectively improving the efficiency and accuracy of vegetable growth monitoring. However, traditional machine learning methods require manual feature selection and the construction of relatively complex models, and their ability to model complex nonlinear relationships is limited. Deep learning algorithms can automatically extract key features from complex crop images and are widely used in leafy vegetable fresh weight estimation.

[0026] The closest implementation to this application is as follows: Invention patent CN116757332 provides a method, device, equipment, and medium for predicting leafy vegetable yield. This method constructs a leafy vegetable yield prediction model based on a mechanistic model and machine vision. It fits the yield predicted by the machine vision-based image segmentation model with the yield predicted by the mechanistic model, which can correct the yield data estimated by a single image algorithm or mechanistic model, thereby improving the estimation accuracy. However, this method requires image segmentation, which will result in the loss of certain image features, and it does not consider the transferability of fresh weight estimation across different leafy vegetable species. Invention patent CN118154855 discloses a method for detecting camouflaged targets of green fruits. It uses MobileVit as the backbone network, integrates features from multiple levels of backbone using multiple edge-guided feature modules, and designs a texture boundary perception module to extract information at different scales and levels to achieve efficient detection of green fruits. However, this method is only used for target detection in images and lacks mapping with real space and measured fresh weight data.

[0027] There are several methods for estimating the fresh weight of leafy vegetables, as detailed below: (1) Traditional weighing method is simple but not suitable for large-scale real-time monitoring.

[0028] (2) Traditional statistical models estimate fresh weight by measuring indicators such as plant height, number of leaves, and canopy area of ​​leafy vegetables, combined with empirical formulas or regression models. This requires complicated manual measurements and depends on historical yield data and simple environmental factors.

[0029] (3) Fresh weight estimation based on machine learning. By collecting phenotypic characteristics and growth environment parameters of leafy vegetables, a model is built using machine learning methods to achieve non-destructive estimation of fresh weight. A predictive model is constructed by analyzing historical data and environmental factors. This method can handle complex nonlinear relationships and can make predictions using meteorological data, soil data, and historical yield data, but it requires complex statistical features and multi-dimensional auxiliary data.

[0030] (4) Deep learning-based methods. Fresh weight is estimated by capturing images of the canopy of leafy vegetables and combining them with deep learning models such as convolutional neural networks (CNNs). Lossless estimation is achieved through automatic feature extraction, image analysis, and regression equations. It has the advantages of being fast, lossless, and suitable for large-scale applications.

[0031] There are still many challenges in estimating the fresh weight of leafy vegetables: 1) Most methods require image segmentation before feature extraction, which is a complex process with some information loss; 2) Even with very complex network design, it is difficult to fully extract image features; 3) The imaging distortion of three-dimensional objects by two-dimensional images leads to phenotypic estimation errors; 4) There is no research on cross-species fresh weight estimation of leafy vegetables.

[0032] Figure 1 This is one of the flowcharts illustrating the fresh weight estimation method for leafy vegetables provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain images of leafy vegetables of the first target type.

[0033] Specifically, to achieve flexible, efficient, and convenient fresh weight estimation of vegetables, this embodiment first acquires images of leafy vegetables of the first target species. Optionally, multiple data acquisitions can be conducted at different growth stages after the leafy vegetables are transplanted. Each acquisition includes single-plant canopy image data acquisition and measured single-plant phenotypic data acquisition. Image data acquisition includes using image acquisition devices such as smartphones and cameras to acquire in-situ top-down RGB photos of the canopy of a certain leafy vegetable at different growth stages. To ensure consistency in the mapping ratio between the image and the real scene, the top-down image acquisition must meet one of the following two conditions: 1) the shooting height is known; 2) a non-green reference card of known area is placed on the ground in the field of view, with a color significantly different from the plant. Phenotypic data acquisition includes destructive sampling of each photographed plant to collect the fresh weight of a single leafy vegetable plant and record the seedling age of the leafy vegetables. The plant phenotypic data corresponds one-to-one with the image data.

[0034] Step 102: Extract the visible light vegetation index, geometric information, and color information from the leafy vegetable images.

[0035] Specifically, in this embodiment, after acquiring images of leafy vegetables, visible light vegetation indices, geometric information, and color information are extracted. In this embodiment, the visible light vegetation index (VI) is an index that assesses vegetation condition by analyzing the reflectance differences in visible light bands (such as red and green light). These indices can reflect various vegetation information such as vegetation cover, growth vitality and health status, chlorophyll content, and biomass. This application extracts multiple vegetation indices from RGB images to quantitatively analyze the dynamic changes in vegetation, in order to establish a more accurate fresh weight estimation model for leafy vegetables. These include, but are not limited to, the vegetation index features shown in Table 1: Table 1

[0036] Note: R, B, and G represent the atmospheric corrected surface reflectance in the red, blue, and green infrared channels, respectively.

[0037] Optionally, geometric features are extracted from RGB images of leafy vegetables by pixel counting, directly obtaining various geometric features such as area, perimeter, convex hull area, convex hull perimeter, circle diameter, minimum area rectangle width, and minimum area rectangle height. These geometric features help the model more comprehensively understand and analyze the attributes of objects in the image, effectively reflecting the biological characteristics and growth status of leafy vegetables.

[0038] Optionally, the mean and standard deviation of different color spaces in color features are important statistics describing the central tendency and dispersion of the color distribution in an image. This invention extracts dozens of color features from RGB images, including the mean and standard deviation of color components across multiple color spaces.

[0039] Step 103: Input the leafy vegetable image, visible light vegetation index, geometric information and color information into the leafy vegetable fresh weight prediction model, and output the prediction result of the leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model is used to estimate the fresh weight of leafy vegetables; the leafy vegetable fresh weight prediction model includes: an initial feature extraction module, a multi-scale feature extraction module and a multi-dimensional feature fusion module; The initial feature extraction module is used to extract initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, and color features from leafy vegetable images, visible light vegetation index, geometric information, and color information. The multi-scale feature extraction module is used to obtain multi-scale features corresponding to leafy vegetable images; The multidimensional feature fusion module is used to fuse the initial leafy vegetable image features, the initial visible light vegetation index features, the initial geometric features and color features, and the multi-scale features corresponding to the leafy vegetable image to obtain the multidimensional feature fusion result and output the estimated result of the fresh weight of the leafy vegetables.

[0040] Specifically, the leafy vegetable fresh weight prediction model in this embodiment includes an initial feature extraction module (IFE), a multi-scale feature extraction module (MSFE), and a multi-dimensional feature fusion module (MIFF). Optionally, as... Figure 2 As shown, the visible light vegetation index (VLAI), represented by a two-dimensional matrix, and the geometric and color information, represented by numerical values, calculated from the RGB image, are processed by the Initial Feature Extraction (IFE) module to obtain initial leafy vegetable image features, initial VLAI features, initial geometric features, and color features. The shallow RGB image features are then used for feature extraction and fusion in the backbone network, undergoing a three-level feature extraction and fusion process consisting of alternating stacked Multi-Scale Feature Extraction (MSFE) and Multi-Dimensional Feature Fusion (MIFF) modules, with each level separated by MaxPooling. The first and third levels each contain two sets of structures alternating between MSFE and MIFF modules, used to fully extract and fuse multi-scale and multi-dimensional features. The VLAI features, initial geometric features, and color features extracted by the IFE module are used as inputs to each MIFF module in each level of the backbone network and effectively fused with the multi-scale RGB features, collectively serving as the basis for determining the fresh weight of leafy vegetables. It is worth noting that before the visible vegetation index features are incorporated into each layer, they need to undergo a convolution with a kernel size of 3×3 and a stride of 2. This reduces the size of the feature map by half to match the size of the feature map in the current layer of the backbone network. After feature extraction and fusion at the three layers, the fresh weight estimation results of leafy vegetables in the current input image are obtained through an average pooling layer, a Flatten layer, a Dropout layer, and a fully connected layer (FC).

[0041] It should be noted that, firstly, this application embodiment is the first to integrate vegetation indices into the fresh weight estimation network model, thereby enabling comprehensive and accurate estimation of leafy vegetable fresh weight based on multiple vegetation information such as coverage, growth vitality and health status, chlorophyll content, and biomass, thus improving the accuracy of the leafy vegetable fresh weight estimation results. Secondly, in the process of estimating the fresh weight of leafy vegetables, this application embodiment integrates the original RGB image of the leafy vegetables, vegetation indices, geometric features, and color features, thus fully utilizing multi-dimensional features to assess the growth status of leafy vegetables, effectively improving the accuracy of the leafy vegetable fresh weight estimation results, and realizing end-to-end leafy vegetable fresh weight estimation.

[0042] The method described in the above embodiments integrates vegetation indices into the fresh weight estimation network model, thereby enabling comprehensive and accurate estimation of the fresh weight of leafy vegetables based on multiple vegetation information such as coverage, growth vitality and health status, chlorophyll content, and biomass, thus improving the accuracy of the fresh weight estimation results. Furthermore, by integrating the original RGB images, vegetation indices, geometric features, and color features of leafy vegetables, it fully utilizes multidimensional features to assess the growth status of leafy vegetables, effectively improving the accuracy of the fresh weight estimation results and achieving end-to-end fresh weight estimation of leafy vegetables.

[0043] In some embodiments, the initial feature extraction module (IFE) includes: The module includes an initial RGB feature extraction module, an initial geometric feature and color feature extraction module, and an initial vegetation index feature extraction module. The initial RGB feature extraction module includes multiple convolutional blocks used to extract initial leafy vegetable image features from leafy vegetable images. The initial geometric and color feature extraction module includes fully connected layers and activation functions, which are used to extract initial geometric and color features from geometric and color information. The initial vegetation index feature extraction module includes multiple convolutional blocks used to extract initial visible vegetation index features from the visible vegetation index.

[0044] Specifically, such as Figure 3 As shown, since the visible light vegetation index, geometric information and color information determined from leafy vegetable images have different types of organization, the initial feature extraction module in this embodiment includes an initial RGB feature extraction module, an initial geometric feature and color feature extraction module, and an initial vegetation index feature extraction module, thereby accurately and efficiently extracting multiple types of features such as RGB images, geometric and color features, and visible light vegetation index.

[0045] Optionally, leafy vegetable images have fewer bands. In this embodiment, an initial RGB feature extraction module including a first convolutional kernel is used for feature extraction. Optionally, the first convolutional kernel can be a large 7×7 convolutional kernel, which can perceive a wider range of image features and effectively reduce the parameters and complexity of the model; at the same time, 1×1 and 3×3 convolutional kernels and MaxPooling are used to reduce the size of the feature map and increase the number of channels to improve the feature representation capability.

[0046] Optionally, in this embodiment, the geometric and color features of each sample are represented in the form of a one-dimensional vector. Therefore, a fully connected layer and ReLU stack are used to extract the initial geometric and color feature representations. These initial features will be used as the shared input of each subsequent MIFF module to improve the execution efficiency of the network.

[0047] Optionally, since the visible vegetation index is represented in the form of a two-dimensional image and has a large number of bands, a 3×3 convolution kernel is chosen instead of a large 7×7 convolution kernel to reduce the parameters. At the same time, multi-scale vegetation index features are extracted by using two parallel branches of convolutional layers and MaxPooling layers to fully ensure that the vegetation index features have a good feature fusion foundation.

[0048] The method in the above embodiments includes an initial RGB feature extraction module, an initial geometric and color feature extraction module, and an initial vegetation index feature extraction module, thereby accurately and efficiently extracting various types of features such as RGB images, geometric and color features, and visible light vegetation indices. Furthermore, for leafy vegetable images with fewer bands, using an initial RGB feature extraction module with a larger convolution kernel allows for the perception of a wider range of image features, effectively reducing model parameters and complexity, and improving feature representation capabilities. For visible light vegetation indices with more bands, using an initial vegetation index feature extraction module with a smaller convolution kernel ensures that the vegetation index features have a good foundation for feature fusion, effectively improving the accuracy of feature extraction and the accuracy of leafy vegetable fresh weight estimation results.

[0049] In some embodiments, the multi-scale feature extraction module includes: Convolutional layers and pooling layers with different kernel sizes are used to expand the features of leafy vegetable images, resulting in multi-scale features corresponding to the leafy vegetable images.

[0050] Specifically, the size of leafy vegetables in an image is a direct reflection of their growth status and fresh weight. To fully extract information about leafy vegetables of different sizes, the multi-scale feature extraction module in this embodiment uses a combination of convolutional layers and pooling layers with different kernel sizes to expand image features. Specifically, three convolutional layers with kernel sizes of 1×1, 3×3, and 5×5, along with one MaxPooling layer, form the multi-scale feature extraction module (MSFE) in parallel. This allows for the extraction of multi-scale features from the feature map, capturing the size of leafy vegetables and ensuring the accuracy of fresh weight estimation results. Figure 4As shown, the 3×3 and 5×5 convolutional layers and the 1×1 convolutional layer added to the MaxPooling branch in this structure can reduce the dimensionality of the input features, thereby reducing computation, and can also help the model learn richer features. Subsequently, a channel-dimensional concatenation strategy is used to concatenate the extracted multi-scale features into a unified feature set along the channel dimension, which is then input into the MIFF module for further feature fusion and integration. That is, in this embodiment, convolutional layers and pooling layers with different kernel sizes expand the image features for each input, realizing the extraction and fusion of multi-scale image features, which are then input into the Multi-Dimensional Image Feature Fusion (MIFF) module for further feature fusion and integration.

[0051] The method in the above embodiments includes a multi-scale feature extraction module comprising convolutional layers and pooling layers with different kernel sizes. On the one hand, it can fully extract multi-scale features from the feature map, capture the size of leafy vegetables, and ensure the accuracy of fresh weight estimation results. On the other hand, it can also reduce the dimensionality of the input features, reduce the amount of computation, help the model learn richer features, and improve the accuracy of fresh weight estimation results for leafy vegetables.

[0052] In some embodiments, the multidimensional feature fusion module includes: The multi-dimensional feature fusion module includes: First fusion block and second fusion block; wherein... The first fusion block includes a fully connected layer, an activation function, and a fully connected layer connected in sequence, which is used to fuse the initial geometric features and color features with the multi-scale features corresponding to the leafy vegetable image to obtain the first fused feature; The second fusion block includes a convolutional layer, an activation function, and another convolutional layer connected in sequence. It is used to fuse the initial visible vegetation index features and the first fusion feature to obtain a multi-dimensional feature fusion result.

[0053] Specifically, the Multidimensional Feature Fusion (MIFF) module in this embodiment can integrate multidimensional features such as vegetation index, geometric features, and color features into an RGB image, and is a core component of the entire network. For example... Figure 5As shown, its internal structure consists of two sequentially connected fusion blocks: a first GC Fusion Block and a second VI Fusion Block. For the input multi-scale features, the first GC Fusion Block first transforms the geometric shape and color features into geometric and color feature enhancement factors to correct the input features. Then, the corrected features are input into the second VI Fusion Block, where the visible vegetation index is transformed into a vegetation index enhancement factor, thus further integrating the vegetation index on top of the previous geometric shape and color feature enhancements. The first GC Fusion Block and the second VI Fusion Block mainly consist of two fully connected layers and two 1×1 convolutional layers, each containing a small number of trainable parameters, enabling effective fusion of multi-dimensional features with only a slight increase in computational burden. The output dimension of the first fully connected layer in the first GC Fusion Block includes a scaling parameter *r*, a hyperparameter that allows for changes in network computational cost, thereby finding a balance between computational cost and performance.

[0054] Optionally, in the ( The first level ( When =1,3 =1,2; When =2, =1,…, The initial shape and color features extracted by the I-GC-FE Block in each MIFF Module. He Yudi 3 levels Vegetation index features extracted from 3 convolutional layers The feature transformation functions in the GC Fusion Block and VI Fusion Block are respectively modeled as feature enhancement factors by parameters. and The dimensions are respectively and (in , , and These are the input multi-scale features. (Height, width, and number of channels). Its feature transformation modeling process can be represented by equations (1) and (2) respectively: (1) (2) In the formula, It consists of two fully connected layers, a ReLU activation function, a Sigmoid activation function, and a Reshape function; Composed of two convolutional kernels of size 1 It consists of a convolutional layer of type 1 and a ReLU activation function. For the first... Each level, the first Multiscale features of MSFE Block output (dimension is) The MIFF Module uses feature enhancement factors. and Perform feature transformation This is done by incorporating geometric shape features, color features, and vegetation index features, respectively. The process can be represented as: (3) in, This indicates an element-wise multiplication operation, which preserves the original shape of the input features. Therefore, the blended features output by the MIFFModule... The dimension is still The MIFF Module can adaptively incorporate multidimensional features into the lettuce fresh weight estimation process, by stacking features multiple times to progressively enhance the beneficial feature representations that contribute to fresh weight estimation.

[0055] It should be noted that, in this embodiment, the first fusion block in the multi-dimensional feature fusion module only includes a fully connected layer, an activation function, and a fully connected layer connected in sequence, and the second fusion block in the multi-dimensional feature fusion module only includes a convolutional layer, an activation function, and a convolutional layer connected in sequence. This allows for the fusion of multi-scale features corresponding to the target geometric features and target color features to obtain a first fused feature. The first fused feature, the multi-scale features corresponding to the target visible light vegetation index feature, and the multi-scale features corresponding to the target RGB image feature are then fused to obtain the multi-dimensional feature fusion result. In other words, in this embodiment, the multi-dimensional feature fusion module effectively integrates three image features—vegetation index, geometric features, and color features—with RGB image features with only a small increase in parameters and computational cost, effectively improving the efficiency of feature fusion and the accuracy of fresh weight estimation results for leafy vegetables.

[0056] The method described in the above embodiments allows the multi-dimensional feature fusion module to effectively integrate three image features—vegetation index, geometric features, and color features—with RGB image features with only a small number of additional parameters and computational costs. This enables the full utilization of multi-dimensional features to assess the growth status of leafy vegetables and effectively improves the accuracy of fresh weight estimation results for leafy vegetables.

[0057] In some embodiments, after acquiring an image of a leafy vegetable of the first target type, the method further includes: The pixel coefficient P corresponding to the leafy vegetable image is fixed; wherein, the pixel coefficient P corresponding to the leafy vegetable image is determined based on the following method: Pixel coefficients of leafy vegetable images .

[0058] Specifically, in this embodiment of the application, after acquiring images of leafy vegetables, the top-down photos taken at different shooting heights and with different reference cards are mapped to images with the same spatial scale by using the pixel coefficient P corresponding to the leafy vegetable images. This can achieve consistency between the mapping scale of each image and the real space, thereby effectively improving the accuracy and comprehensiveness of feature extraction and feature fusion, and improving the accuracy of the fresh weight estimation results of leafy vegetables.

[0059] For example, leafy vegetable image data can be preprocessed, such as plant pixel correction and image size normalization. Optionally, phenotypic data preprocessing can also be performed, that is, cleaning the collected leafy vegetable plant data, removing missing values, and standardizing the data.

[0060] Furthermore, by using a fixed pixel coefficient P, top-down photos taken at different shooting heights and with different reference cards can be mapped to images of the same spatial scale, achieving consistency in the mapping scale between each image and the real space, thereby effectively improving the accuracy of feature extraction and feature fusion. Specifically, the pixel coefficients corresponding to leafy vegetable images are determined based on the following method: .

[0061] The method described in the above embodiments, after acquiring images of leafy vegetables, can achieve consistency in the mapping scale between each image and the real space by fixing pixel coefficients, thereby effectively improving the accuracy of feature extraction and feature fusion as well as the accuracy of the fresh weight estimation results of leafy vegetables.

[0062] In some embodiments, after training the leafy vegetable fresh weight prediction model, the method further includes: The fresh weight of the second target type of leafy vegetables was estimated using the leafy vegetable fresh weight prediction model, and the fresh weight estimation results of the second target type of leafy vegetables were obtained.

[0063] Specifically, in this embodiment, several other leafy vegetables of different types than those used in the model construction are selected. For example, the model construction uses lettuce as the first target type, while the type transfer uses other leafy vegetables such as spinach, celery, and bok choy as the second target type. More than 30 sets of top-view and fresh weight data of a single plant are collected for each leafy vegetable. The processed top-view data are input into the leafy vegetable fresh weight prediction model trained in this application to obtain the predicted fresh weight value per plant. The predicted value is then fitted with the true value to obtain the transfer function for that type. This process is repeated to obtain the transfer function for each leafy vegetable, resulting in a model transfer function table for different leafy vegetable types. This allows for efficient and accurate estimation of fresh weight across different types of leafy vegetables. Optionally, in-situ growth and planting images of multiple leafy vegetable types can be collected, and a common convolutional neural network (CNN) classification network can be used to construct a leafy vegetable type recognition module. This allows for accurate identification of the leafy vegetable type based on the input image.

[0064] The method in the above embodiment estimates the fresh weight of a second target type of leafy vegetables based on the leafy vegetable fresh weight prediction model, and obtains the fresh weight estimation result of the second target type of leafy vegetables. Based on the fresh weight estimation result of the second target type of leafy vegetables and the fresh weight measurement result of the second target type of leafy vegetables, a species transfer function is generated, which can efficiently and accurately realize the fresh weight estimation of leafy vegetables across species, effectively expanding the application scenario of the leafy vegetable fresh weight prediction model in this application and realizing accurate estimation of the fresh weight of different types of leafy vegetables.

[0065] In some embodiments, based on images of leafy vegetables, visible light vegetation index, geometric information, and color information are extracted, including: Based on images of leafy vegetables, various visible light vegetation indices, geometric information, and color information were determined. Correlation analysis was performed on various visible light vegetation indices, geometric information, color information, and the measured fresh weight of leafy vegetables to obtain the selected visible light vegetation indices, geometric information, and color information.

[0066] Specifically, as shown in Table 1, in this embodiment of the application, various visible light vegetation indices, geometric information, and color information can be determined based on images of leafy vegetables. Further, in this implementation, various geometric and color information are summarized, and correlation analysis is performed with measured fresh weight using methods such as the Pearson coefficient. Feature filtering is then performed according to a certain correlation threshold to obtain the filtered visible light vegetation indices, geometric information, and color information. Therefore, utilizing the filtered visible light vegetation indices, geometric features, and color information can effectively improve the efficiency and accuracy of feature extraction and feature fusion, while reducing the computational load and complexity of feature extraction and feature fusion. For example, the filtered visible light vegetation indices, geometric information, and color information are shown in Table 2.

[0067] Table 2

[0068] The method described above summarizes various geometric and color features extracted from RGB images and performs correlation analysis with measured fresh weight. Feature filtering is performed according to a certain correlation threshold to obtain the filtered visible light vegetation index, geometric features, and color features. This can effectively improve the efficiency and accuracy of feature extraction and feature fusion, and reduce the computational load and complexity of feature extraction and feature fusion.

[0069] For example, such as Figure 6 and Figure 7 As shown in the figure, this application provides a method for estimating the fresh weight of leafy vegetables, and the specific process is as follows: Users take and upload top-view RGB images of leafy vegetables that meet the requirements using their mobile phones or other image acquisition terminals. The system first calls the leafy vegetable type recognition module to automatically determine the type of leafy vegetable in the photo, or to have it manually confirmed. Based on the identified or confirmed type of leafy vegetable, the system determines the model transfer function f for that type of leafy vegetable.

[0070] Secondly, after normalizing the captured images, they are input into the leafy vegetable fresh weight prediction model to predict the fresh weight.

[0071] Finally, the predicted fresh weight is corrected using the model transfer function f for this type of leafy vegetable to obtain the final estimated fresh weight value for this type of leafy vegetable.

[0072] Optionally, the leafy vegetable fresh weight prediction model is trained in the following manner: The dataset was divided into training, validation, and test sets in a 6:2:2 ratio for network training, validation, and testing, respectively. The training set was augmented using the data augmentation method described in section 2.1.3, increasing the original data size by 10 times. The data partitioning and sample size of each dataset were clearly defined, and all experimental results were statistically analyzed on the test set. The Adam optimizer was used for training, with the learning rate, batch size, and epochs set to 0.0001, 32, and 200, respectively, and the MSE loss function chosen. In the proposed method, N and... The parameters were set to 3 and 16 respectively. The models were trained without any pre-training parameters to fully evaluate the performance of the proposed networks. Sixteen algorithms were used for comparison, including eight classic convolutional neural networks, two self-built models, and six traditional machine learning algorithms. Among them, the seven classic convolutional neural networks were trained using the Adam optimizer, with learning rate, batch size, and epochs set to 0.0001, 32, and 200 respectively, and MSE as the loss function. All classic convolutional neural network models were trained using pre-training parameters. The CNN_Xu model was trained using the Adam optimizer, with learning rate, batch size, and epochs set to 0.0001, 128, and 300 respectively, and MSPE as the loss function. The CNN_zhang model was trained using the SGD optimizer, with an initial learning rate of 0.001, decreasing by a factor of 0.1 every 20 epochs, batch size and epochs set to 128 and 300 respectively, and MSE as the loss function. This model was trained without any pre-training parameters. The DNN network has 5 hidden layers and is trained using the Adam optimizer with a learning rate, batch size, and epochs set to 0.0001, 32, and 200, respectively. The loss function is MSE. Five traditional machine learning models are trained using default parameters.

[0073] For example, to accurately evaluate the model's performance, this application uses the coefficient of determination (R²), root mean square error (RMSE), and mean absolute error (MAE) as evaluation metrics. These three metrics are also commonly used to rate model performance in regression tasks of machine learning. R² represents the model's ability to explain data variability; the closer its value is to 1, the better the model fits, meaning the model can explain more data variability, as defined in formula (4). RMSE measures the average error between predicted and actual values, as defined in formula (5). MAE calculates the average of the absolute values ​​of the differences between all predicted and actual values, as defined in formula (6).

[0074] (4) (5) (6) Optionally, to verify the effectiveness of the proposed network, experiments were conducted on one self-built dataset and two publicly available datasets. The performance of the proposed network was comprehensively compared with that of eight classic convolutional neural networks, two existing lettuce fresh weight estimation models, and six machine learning models on three lettuce fresh weight estimation datasets. Simultaneously, the important role of the multi-dimensional feature fusion approach proposed in this invention in the lettuce fresh weight estimation task was evaluated.

[0075] To highlight the advantages of the proposed model in lettuce fresh weight estimation, we compared it with 16 algorithms, including 8 classic convolutional neural networks (Xception, Densenet121, Resnet50, Resnet34, GoogLeNet, VGG16, ConvNext, and AlexNet), 2 self-built models (Zhang et al., 2020 and Xu et al., 2023), and 6 traditional machine learning algorithms (Random Forest, Extra Trees, Gradient Boosting, XGBoost, AdaBoost, and DNN). Among these, Zhang et al., 2020 and Xu et al., 2023 are the algorithms proposed for handling lettuce fresh weight estimation. Lettuce datasets have diverse sample types and exhibit significant differences in sample size, shape, and texture. Therefore, we selected two public lettuce datasets for cross-dataset evaluation of MIFFNet's generalization ability. We also trained and tested 16 comparative models and our proposed model on two public datasets, and compared and analyzed the models using three evaluation metrics: RMSE, MAE, and R2.

[0076] In the lettuce fresh weight estimation task, a comparison of model training results based on three datasets shows that the proposed model method performs exceptionally well in both DatasetA and DatasetB, with R² values ​​as high as 0.940 and 0.943 respectively. The RMSE and MAE values ​​are the lowest among all models, indicating that the proposed model structure has stable performance and good generalization ability, demonstrating stable and good performance across different datasets. While traditional CNNs have the potential for high accuracy in image-based predictions, they may not be the most suitable models for lettuce fresh weight estimation tasks. Specific models need to be built for task-specific problems, while also possessing good generalization ability and stable performance. The MIFFNet model proposed in this invention exhibits superior performance, demonstrating its potential as a valuable tool for real-time, on-site estimation of lettuce fresh weight. It fully showcases its enormous potential and advantages in lettuce fresh weight estimation, accurately and reliably estimating lettuce fresh weight.

[0077] In summary, this invention proposes an end-to-end fresh weight estimation network and method for leafy vegetables that integrates RGB images, vegetation indices, geometric features, and color features. Multi-dimensional features are fed into the network for fresh weight estimation, and for the first time, vegetation indices are integrated into the fresh weight estimation model. In the structural design, the model cleverly employs alternating stacks of multi-scale feature extraction modules and multi-dimensional feature fusion modules to form the backbone network of this method, enhancing the model's ability to capture features at different scales. In the image processing stage, all overhead images are mapped to a fixed spatial scale using a fixed pixel coefficient P. This method is flexible and convenient, while also improving image standardization. A leafy vegetable species identification module is introduced, which can automatically determine the category of the photographed leafy vegetables based on the image. A leafy vegetable species transfer function calculation module is constructed, which can calculate the model transfer function for each leafy vegetable species based on a small amount of test data, providing parameters for cross-species fresh weight estimation.

[0078] It should be noted that the method in this application introduces several innovations in design and implementation, effectively addressing the limitations of existing technologies and providing a more comprehensive and accurate estimation of the fresh weight of leafy vegetables. Data acquisition is simple, requiring only the acquisition of an RGB overhead image and fresh weight data of a single leafy vegetable plant. By fixing pixel coefficients, consistency in the mapping scale between the image and the real space is achieved. An end-to-end network method for estimating the fresh weight of leafy vegetables is implemented, fusing the original RGB image, vegetation index, geometric features, and color features. Furthermore, cross-species fresh weight estimation of leafy vegetables is achieved through a leafy vegetable species transfer function.

[0079] Comparative experiments were conducted on three different lettuce datasets, comparing the proposed method with over ten algorithms, including classic convolutional neural networks, existing lettuce fresh weight estimation models, and traditional machine learning algorithms. Experimental results show that the proposed method performs well on all three datasets. On the self-built dataset, the R² reached 0.929, with RMSE and MAE of 28.544 and 14.446, respectively. On the two public datasets, the R² reached 0.94 and 0.943, respectively, and the RMSE and MAE were also lower than other comparative models. It significantly outperforms baseline methods that do not integrate multidimensional features and all other comparative methods (including classic convolutional neural networks pre-trained on ImageNet). Simultaneously, the number of parameters is reduced by 41.5%. Extensive comparative experiments demonstrate that the proposed network has the advantages of low computational cost, lightweight design, and high accuracy. Compared with other models, this method is more advantageous in the task of estimating the fresh weight of leafy vegetables.

[0080] The following describes the leafy vegetable fresh weight estimation device provided by the present invention. The leafy vegetable fresh weight estimation device described below can be referred to in correspondence with the leafy vegetable fresh weight estimation method described above. The leafy vegetable fresh weight estimation device of this application embodiment is as follows: Figure 8 As shown, it includes: The acquisition module 810 is used to acquire images of leafy vegetables of the first target type; The determination module 820 is used to extract visible vegetation index, geometric information and color information based on leafy vegetable images; The estimation module 830 is used to input leafy vegetable images, visible light vegetation index, geometric information and color information into the leafy vegetable fresh weight prediction model, and output the prediction result of leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model is used to estimate the fresh weight of leafy vegetables; the leafy vegetable fresh weight prediction model includes: an initial feature extraction module, a multi-scale feature extraction module and a multi-dimensional feature fusion module.

[0081] Figure 9 A schematic diagram of the physical structure of an electronic device is provided. This electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940. The processor 910, communication interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a leafy vegetable fresh weight estimation method. This method includes: acquiring an image of a first target type of leafy vegetable; extracting visible light vegetation index, geometric information, and color information from the leafy vegetable image; inputting the leafy vegetable image, visible light vegetation index, geometric information, and color information into a leafy vegetable fresh weight prediction model; and outputting a prediction result of the leafy vegetable fresh weight. The leafy vegetable fresh weight prediction model is used to estimate the fresh weight of leafy vegetables. The leafy vegetable fresh weight prediction model includes: an initial feature extraction module, a multi-scale feature extraction module, and a multi-dimensional feature fusion module. The system comprises three modules: an initial feature extraction module, an initial visible light vegetation index (VLAI) feature extraction module, an initial geometric feature extraction module, and a color feature extraction module, and a multi-scale feature fusion module. The initial feature extraction module extracts the initial leafy vegetable image features, initial VLAI features, initial geometric features, color features, and multi-scale features corresponding to the leafy vegetable image to obtain the multi-scale feature fusion result and outputs the estimated fresh weight of the leafy vegetables.

[0082] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the leafy vegetable fresh weight estimation method provided by the above methods. The method includes: acquiring an image of a first target type of leafy vegetable; extracting visible light vegetation index, geometric information, and color information from the leafy vegetable image; inputting the leafy vegetable image, visible light vegetation index, geometric information, and color information into a leafy vegetable fresh weight prediction model, and outputting a prediction result of the leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model is used for estimating the fresh weight of leafy vegetables; leafy vegetables The fresh weight prediction model includes an initial feature extraction module, a multi-scale feature extraction module, and a multi-dimensional feature fusion module. The initial feature extraction module extracts initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, and color features from leafy vegetable images, visible light vegetation index, geometric information, and color information. The multi-scale feature extraction module obtains the multi-scale features corresponding to the leafy vegetable images. The multi-dimensional feature fusion module fuses the initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, color features, and the multi-scale features corresponding to the leafy vegetable images to obtain the multi-dimensional feature fusion result and outputs the estimated fresh weight of the leafy vegetables.

[0084] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the leafy vegetable fresh weight estimation method provided by the methods described above. This method includes: acquiring an image of a first target type of leafy vegetable; extracting visible light vegetation index, geometric information, and color information from the leafy vegetable image; inputting the leafy vegetable image, visible light vegetation index, geometric information, and color information into a leafy vegetable fresh weight prediction model, and outputting a prediction result of the leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model is used for estimating the fresh weight of leafy vegetables; the leafy vegetable fresh weight prediction model includes: initial feature extraction... The system comprises an initial feature extraction module, a multi-scale feature extraction module, and a multi-dimensional feature fusion module. The initial feature extraction module extracts initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, and color features from leafy vegetable images, visible light vegetation index, geometric information, and color information. The multi-scale feature extraction module acquires the multi-scale features corresponding to the leafy vegetable images. The multi-dimensional feature fusion module fuses the initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, color features, and the multi-scale features corresponding to the leafy vegetable images to obtain the multi-dimensional feature fusion result and outputs the estimated fresh weight of the leafy vegetables.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for estimating the fresh weight of leafy vegetables, characterized in that, include: Obtain images of leafy vegetables of the first target type; Based on the leafy vegetable images, the visible light vegetation index, geometric information, and color information are extracted; The leafy vegetable image, visible light vegetation index, geometric information, and color information are input into the leafy vegetable fresh weight prediction model, which outputs the prediction result of the leafy vegetable fresh weight. The leafy vegetable fresh weight prediction model is used to estimate the fresh weight of leafy vegetables. The leafy vegetable fresh weight prediction model includes: an initial feature extraction module, a multi-scale feature extraction module, and a multi-dimensional feature fusion module. The initial feature extraction module is used to extract initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, and color features from the leafy vegetable image, visible light vegetation index, geometric information, and color information. The multi-scale feature extraction module is used to obtain the multi-scale features corresponding to the leafy vegetable image; The multidimensional feature fusion module is used to fuse the initial leafy vegetable image features, the initial visible light vegetation index features, the initial geometric features and color features, and the multi-scale features corresponding to the leafy vegetable image to obtain the multidimensional feature fusion result and output the estimated result of the fresh weight of the leafy vegetables.

2. The method for estimating the fresh weight of leafy vegetables according to claim 1, characterized in that, The initial feature extraction module includes: The module includes an initial RGB feature extraction module, an initial geometric feature and color feature extraction module, and an initial vegetation index feature extraction module. The initial RGB feature extraction module includes multiple convolutional blocks for extracting initial leafy vegetable image features from the leafy vegetable image. The initial geometric and color feature extraction module includes a fully connected layer and an activation function, used to extract the initial geometric and color features from the geometric and color information; The initial vegetation index feature extraction module includes multiple convolutional blocks for extracting the initial visible light vegetation index features from the visible light vegetation index.

3. The method for estimating the fresh weight of leafy vegetables according to claim 2, characterized in that, The multi-scale feature extraction module includes: Convolutional layers and pooling layers with different kernel sizes are used to expand the features of the leafy vegetable image to obtain multi-scale features corresponding to the leafy vegetable image.

4. The method for estimating the fresh weight of leafy vegetables according to claim 3, characterized in that, The multi-dimensional feature fusion module includes: First fusion block and second fusion block; wherein... The first fusion block includes a fully connected layer, an activation function, and a fully connected layer connected in sequence, used to fuse the initial geometric features and color features with the multi-scale features corresponding to the leafy vegetable image to obtain the first fusion feature; The second fusion block includes a convolutional layer, an activation function, and another convolutional layer connected in sequence, used to fuse the initial visible vegetation index feature and the first fusion feature to obtain a multi-dimensional feature fusion result.

5. The method for estimating the fresh weight of leafy vegetables according to any one of claims 1-4, characterized in that, After acquiring the image of the first target type of leafy vegetable, the process further includes: The pixel coefficient P corresponding to the leafy vegetable image is fixed; wherein, the pixel coefficient P corresponding to the leafy vegetable image is determined based on the following method: Pixel coefficients of leafy vegetable images .

6. The method for estimating the fresh weight of leafy vegetables according to any one of claims 1-4, characterized in that, After training the leafy vegetable fresh weight prediction model, the method further includes: The fresh weight of the second target type of leafy vegetables is estimated based on the leafy vegetable fresh weight prediction model, and the fresh weight estimation result of the second target type of leafy vegetables is obtained. A species transfer function is generated based on the estimated fresh weight of the leafy vegetables of the second target species and the measured fresh weight of the leafy vegetables of the second target species; the species transfer function is used to realize the fresh weight estimation of leafy vegetables across species.

7. The method for estimating the fresh weight of leafy vegetables according to any one of claims 1-4, characterized in that, The step of extracting visible vegetation index, geometric information, and color information from the leafy vegetable image includes: Based on the images of leafy vegetables, various visible light vegetation indices, geometric information, and color information were determined; Correlation analysis was performed on the various visible vegetation indices, geometric information, color information, and the measured fresh weight of the leafy vegetables to obtain the selected visible vegetation indices, geometric information, and color information.

8. A device for estimating the fresh weight of leafy vegetables, characterized in that, include: The acquisition module is used to acquire images of leafy vegetables of the first target type; The determination module is used to extract visible vegetation index, geometric information, and color information based on the leafy vegetable image; The estimation module is used to input the leafy vegetable image, visible light vegetation index, geometric information and color information into the leafy vegetable fresh weight prediction model, and output the prediction result of the leafy vegetable fresh weight; the leafy vegetable fresh weight prediction model is used to estimate the fresh weight of leafy vegetables; the leafy vegetable fresh weight prediction model includes: an initial feature extraction module, a multi-scale feature extraction module and a multi-dimensional feature fusion module; The initial feature extraction module is used to extract initial leafy vegetable image features, initial visible light vegetation index features, initial geometric features, and color features from the leafy vegetable image, visible light vegetation index, geometric information, and color information. The multi-scale feature extraction module is used to obtain the multi-scale features corresponding to the leafy vegetable image; The multidimensional feature fusion module is used to fuse the initial leafy vegetable image features, the initial visible light vegetation index features, the initial geometric features and color features, and the multi-scale features corresponding to the leafy vegetable image to obtain the multidimensional feature fusion result and output the estimated result of the fresh weight of the leafy vegetables.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the leafy vegetable fresh weight estimation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the fresh weight estimation method for leafy vegetables as described in any one of claims 1 to 7.

Images