Method and device for determining fresh weight of lettuce and electronic equipment
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA AGRI UNIV
- Filing Date
- 2024-05-22
- Publication Date
- 2026-08-07
AI Technical Summary
[0005]本发明提供一种生菜的鲜重确定方法、装置及电子设备，用以解决现有技术中确定生菜的鲜重的准确性差的缺陷，实现提高确定生菜的鲜重的准确性
[0016]本发明提供的生菜的鲜重确定方法、装置及电子设备，通过获取待检测生菜的多模态图像，多模态图像包括RGB图像和深度图像；将预处理后的多模态图像输入至少一个融合特征模型，获取融合特征模型输出的融合特征信息；将融合特征信息和多模态图像输入鲜重预测模型，获取鲜重预测模型输出的待检测生菜的鲜重；其中，融合特征模型是在第一预设卷积神经网络模型的基础上，基于样本多模态图像训练得到的；鲜重预测模型是在第二预设卷积神经网络模型的基础上，基于样本融合特征信息、样本多模态图像和样本鲜重的标签训练得到的，第二预设卷积神经网络模型的单模态特征提取器是基于融合特征模型的编码器构建的，鲜重预测模型基于融合特征信息和多模态图像，生成注意力特征图，基于注意力特征图，生成多模态特征图，将多模态特征图映射为鲜重，注意力特征图包括空间注意力特征图和通道注意力特征图。本发明通过融合特征模型和鲜重预测模型实现了对生菜的RGB图像和深度图像的准确融合，有利于提高获取生菜鲜重的准确性。通过鲜重预测模型自动获取鲜重，利用了人工智能技术进行检测，自动化程度高，能够有效减少设施生菜栽培管理的人工干预，降低应用成本和复杂程度，有效提高设施生菜鲜重估算的准确性和实时性。
Smart Images
Figure CN118570793B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of agricultural data processing technology, and in particular to a method, apparatus and electronic equipment for determining the fresh weight of lettuce. Background Technology
[0002] Due to the short growth cycle of greenhouse lettuce, accurately obtaining its growth parameters and understanding its growth status is of great practical significance for farmers to take timely and correct cultivation and management measures to improve the yield and quality of greenhouse lettuce. Fresh weight is a key parameter characterizing the life activities and growth status of greenhouse lettuce, and it is also directly related to its yield. Therefore, estimating the fresh weight of greenhouse lettuce is of great importance for growth monitoring and precision management. Traditional methods for obtaining growth parameters mainly rely on manual measurement. Although these methods can yield relatively reliable results, they are inefficient, consume a lot of time and resources, are destructive, and cause irreversible damage, making them unsuitable for large-scale crop phenotypic information acquisition.
[0003] Currently, computer vision-based methods are widely used for estimating the fresh weight of greenhouse lettuce. However, as a parameter with 3D characteristics, fresh weight is difficult to estimate accurately using only 2D visible light images.
[0004] In summary, the accuracy of existing methods for determining the fresh weight of lettuce is poor. Summary of the Invention
[0005] This invention provides a method, apparatus, and electronic device for determining the fresh weight of lettuce, in order to solve the problem of poor accuracy in determining the fresh weight of lettuce in the prior art, and to improve the accuracy of determining the fresh weight of lettuce.
[0006] In a first aspect, the present invention provides a method for determining the fresh weight of lettuce, comprising: acquiring a multimodal image of the lettuce to be detected, the multimodal image including an RGB image and a depth image; inputting the preprocessed multimodal image into at least one fusion feature model to obtain fusion feature information output by the fusion feature model; inputting the fusion feature information and the multimodal image into a fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model; wherein, the fusion feature model is trained based on the sample multimodal image on the basis of a first preset convolutional neural network model; the fresh weight prediction model is trained based on the sample fusion feature information, the sample multimodal image and the label of the sample fresh weight on the basis of a second preset convolutional neural network model, the single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model, the fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal image, generates a multimodal feature map based on the attention feature map, maps the multimodal feature map to fresh weight, and the attention feature map includes a spatial attention feature map and a channel attention feature map.
[0007] According to the method for determining the fresh weight of lettuce provided by the present invention, the encoder of the fusion feature model includes an RGB feature encoder and a deep feature encoder. The fresh weight prediction model is obtained based on the following steps: using the RGB feature encoder as the RGB single-modal feature extractor of the second preset convolutional neural network model, and using the deep feature encoder as the deep single-modal feature extractor of the second preset convolutional neural network model; constructing the second preset convolutional neural network model based on the input layer, the RGB single-modal feature extractor, the deep single-modal feature extractor, the multimodal feature fusion module, and the output layer; using the sample fresh weight as the third label, obtaining the third training sample based on the third label, the sample multimodal image, and the matching sample fusion feature information, where the matching sample fusion feature information is the sample fusion feature information that matches the sample multimodal image; and training the second preset convolutional neural network model based on the third training sample to obtain the fresh weight prediction model.
[0008] According to the method for determining the fresh weight of lettuce provided by the present invention, the sample multimodal image includes a sample depth image and a sample RGB image, the fusion feature model includes an RGB fusion feature model and a depth fusion feature model, the first preset convolutional neural network model includes a preset RGB model and a preset depth model, and the fusion feature model is obtained based on the following steps: using the sample depth image as the first label, obtaining a first training sample based on the first label and the sample RGB image, training the preset RGB model based on the first training sample to obtain the RGB fusion feature model; using the sample RGB image as the second label, obtaining a second training sample based on the second label and the sample depth image, training the preset depth model based on the second training sample to obtain the depth fusion feature model.
[0009] According to the method for determining the fresh weight of lettuce provided by the present invention, the fused feature information includes multiple RGB feature maps and multiple depth feature maps. The fused feature model is used to generate the fused feature information: a depth feature encoder based on the RGB fusion feature model sequentially performs multiple depth feature extractions on the RGB image, and obtains a depth feature map based on the result of each depth feature extraction, so as to obtain multiple depth feature maps; an RGB feature extractor based on the depth fusion feature model sequentially performs multiple RGB feature extractions on the depth image, and obtains an RGB feature map based on the result of each RGB feature extraction, so as to obtain multiple RGB feature maps.
[0010] According to the fresh weight determination method for lettuce provided by the present invention, the spatial attention feature map includes an RGB spatial attention feature map and a depth spatial attention feature map. The fresh weight prediction model is used to generate the spatial attention feature map: multiple RGB feature maps are numbered according to the feature extraction order; the last RGB feature map is subjected to average pooling to obtain a first average feature; the last RGB feature map is subjected to max pooling to obtain a first maximum feature; the first maximum feature and the first average feature are concatenated, convolved, and activated to obtain a first weight; the RGB spatial attention feature map is obtained based on the first weight and the last RGB image. The last RGB image is obtained by convolutional feature extraction based on the previous RGB single-modal feature map, and the previous RGB single-modal feature map is the sum of the previous RGB channel attention feature map and the previous RGB spatial attention feature map. The resulting RGB spatial attention feature map is the feature map preceding the RGB spatial attention feature map. Multiple depth feature maps are numbered based on the feature extraction order. Average pooling is performed on the last depth feature map to obtain the second average feature. Max pooling is then performed on the last depth feature map to obtain the second maximum feature. The second maximum feature and the second average feature are concatenated, convolved, and activated to obtain the second weight. The depth spatial attention feature map is obtained based on the second weight and the last depth image. The last depth image is obtained by convolutional feature extraction based on the previous depth unimodal feature map. The previous depth unimodal feature map is obtained by summing the previous depth channel attention feature map and the previous depth spatial attention feature map. The previous depth spatial attention feature map is the feature map preceding the depth spatial attention feature map.
[0011] According to the fresh weight determination method for lettuce provided by the present invention, the channel attention feature map includes an RGB channel attention feature map and a depth channel attention feature map. The fresh weight prediction model is used to generate the channel attention feature map: coordinate attention is extracted from the last RGB image to obtain an initial RGB channel attention feature map, and coordinate attention is extracted from the last depth image to obtain an initial depth channel attention feature map; based on the fusion result of the initial RGB channel attention feature map and the initial depth channel attention feature map, the RGB channel attention feature map and the depth channel attention feature map are obtained.
[0012] According to the fresh weight determination method for lettuce provided by the present invention, the fresh weight prediction model is used to generate multimodal feature maps: based on the sum of RGB channel attention feature maps and RGB spatial attention feature maps, an RGB single-modal feature map is obtained; the RGB single-modal feature map is then subjected to convolution and global average pooling to obtain a target RGB feature map; based on the sum of depth channel attention feature maps and depth spatial attention feature maps, a depth single-modal feature map is obtained; the depth single-modal feature map is then subjected to convolution and global average pooling to obtain a target depth feature map; the target RGB feature map and the target depth feature map are then concatenated and convolved to obtain a multimodal feature map.
[0013] According to the method for determining the fresh weight of lettuce provided by the present invention, the preprocessed multimodal image is obtained based on the following steps: cropping the multimodal image to obtain a cropped image, the cropped image being centered on the lettuce to be detected and including the complete lettuce to be detected; and standardizing the cropped image based on the cropped image, the minimum pixel value of the lettuce region of the cropped image, and the maximum pixel value of the lettuce region of the cropped image to obtain the preprocessed multimodal image, the lettuce region being the image region including the lettuce to be detected.
[0014] Secondly, the present invention provides a device for determining the fresh weight of lettuce, comprising: a multimodal image acquisition module for acquiring multimodal images of the lettuce to be detected, the multimodal images including RGB images and depth images; a fusion feature information determination module for inputting the preprocessed multimodal images into at least one fusion feature model to obtain fusion feature information output by the fusion feature model; and a fresh weight prediction module for inputting the fusion feature information and the multimodal images into a fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model; wherein, the fusion feature model is trained based on a first preset convolutional neural network model and based on sample multimodal images; the fresh weight prediction model is trained based on a second preset convolutional neural network model and based on sample fusion feature information, sample multimodal images, and labels of sample fresh weights, wherein the single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model, the fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal images, generates a multimodal feature map based on the attention feature map, maps the multimodal feature map to fresh weight, and the attention feature map includes a spatial attention feature map and a channel attention feature map.
[0015] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fresh weight determination method of any of the above-described lettuce.
[0016] The present invention provides a method, apparatus, and electronic device for determining the fresh weight of lettuce. The method involves acquiring a multimodal image of the lettuce to be detected, including an RGB image and a depth image; inputting the preprocessed multimodal image into at least one fusion feature model to obtain fusion feature information output by the fusion feature model; and inputting the fusion feature information and the multimodal image into a fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model. The fusion feature model is trained based on a first preset convolutional neural network model using sample multimodal images. The fresh weight prediction model is trained based on a second preset convolutional neural network model using sample fusion feature information, sample multimodal images, and labels of sample fresh weight. The single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model. The fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal image, generates a multimodal feature map based on the attention feature map, and maps the multimodal feature map to fresh weight. The attention feature map includes a spatial attention feature map and a channel attention feature map. This invention achieves accurate fusion of RGB and depth images of lettuce by integrating a feature model and a fresh weight prediction model, which improves the accuracy of lettuce fresh weight estimation. The fresh weight prediction model automatically obtains the fresh weight using artificial intelligence technology, resulting in a high degree of automation. This effectively reduces manual intervention in greenhouse lettuce cultivation and management, lowers application costs and complexity, and significantly improves the accuracy and real-time performance of greenhouse lettuce fresh weight estimation. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is one of the flowcharts illustrating the method for determining the fresh weight of lettuce provided by the present invention;
[0019] Figure 2 This is the second flowchart illustrating the method for determining the fresh weight of lettuce provided by the present invention;
[0020] Figure 3 This is a schematic diagram of the training process of the fusion feature model and freshness prediction model provided by the present invention;
[0021] Figure 4 This is a schematic diagram of the structure of the fresh weight determination device for lettuce provided by the present invention;
[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0024] The following is combined Figures 1-5 This invention describes the method, apparatus, and electronic equipment for determining the fresh weight of lettuce provided in embodiments of the present invention.
[0025] Figure 1 This is one of the flowcharts illustrating the method for determining the fresh weight of lettuce provided by the present invention, such as... Figure 1 As shown, the method for determining the fresh weight of lettuce includes steps S100 to S300, and the specific steps are as follows:
[0026] S100: Acquire a multimodal image of the lettuce to be detected.
[0027] Multimodal images include RGB images and depth images.
[0028] It should be noted that the execution subject of the embodiments of this application can be a server, computer equipment, such as a mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.
[0029] The lettuce to be detected in this embodiment of the invention includes greenhouse lettuce. Depth images can effectively characterize the vertical structure information of greenhouse lettuce, and further fusion with RGB images (red, green, and blue images) can improve the accuracy of fresh weight estimation. However, in different application scenarios, the contributions of RGB images and depth images to fresh weight estimation differ. Existing equivalent stacking fusion methods ignore the complementary information between RGB images and depth images, failing to achieve effective mining and utilization of multimodal images.
[0030] like Figure 1 As shown, multimodal images of lettuce transplanted in greenhouses were obtained 14 days later. The multimodal images include RGB images and depth images, with each RGB image or each depth image containing only one lettuce plant.
[0031] The preprocessed multimodal image is obtained based on the following steps: the multimodal image is cropped to obtain a cropped image, which is centered on the lettuce to be detected and includes the complete lettuce; based on the cropped image, the minimum pixel value of the lettuce region in the cropped image, and the maximum pixel value of the lettuce region in the cropped image, the cropped image is normalized to obtain the preprocessed multimodal image, where the lettuce region is the image region that includes the lettuce to be detected.
[0032] Image preprocessing includes: cropping the multimodal image centered on the lettuce to ensure that the cropped multimodal image contains complete lettuce information, resulting in a cropped image. The cropped image is then standardized, with the following steps:
[0033]
[0034] Where x' is the preprocessed multimodal image, x is the multimodal image, and min(x) l ) represents the minimum pixel value of the lettuce region in the cropped image, max(x) l ) represents the maximum pixel value of the lettuce region in the cropped image.
[0035] Furthermore, the size of the preprocessed multimodal image is adjusted, for example, the size of the preprocessed multimodal image is adjusted to 224 pixels × 224 pixels.
[0036] S200: Input the preprocessed multimodal image into at least one fusion feature model and obtain the fusion feature information output by the fusion feature model.
[0037] The fused feature information includes multiple RGB feature maps and multiple depth feature maps. The fused feature model is used to generate the fused feature information: the depth feature encoder based on the RGB fusion feature model performs multiple depth feature extractions on the RGB image in sequence, and obtains a depth feature map based on the result of each depth feature extraction, so as to obtain multiple depth feature maps; the RGB feature encoder based on the depth fusion feature model performs multiple RGB feature extractions on the depth image in sequence, and obtains an RGB feature map based on the result of each RGB feature extraction, so as to obtain multiple RGB feature maps.
[0038] The fusion feature model includes the RGB fusion feature model and the depth fusion feature model. For example, there are four depth feature maps. Based on the RGB fusion feature model, the first depth feature extraction is performed on the RGB image to obtain the first depth feature map (e.g., size 56×56×64, where 56 is the height, 56 is the width, and 64 is the number of channels or features). Based on the first depth feature map, the second depth feature extraction is performed sequentially to obtain the second depth feature map (e.g., size 56×56×64), and so on, to obtain the third depth feature map (e.g., size 28×28×128) and the fourth depth feature map (e.g., size 14×14×256).
[0039] For example, there are four RGB feature maps. The first RGB feature extraction is performed on the depth image using a deep fusion feature model, resulting in the first RGB feature map (e.g., size 112×112×16). Based on the first RGB feature map, a second RGB feature extraction is performed, resulting in the second RGB feature map (e.g., size 28×28×24), and so on, resulting in the third RGB feature map (e.g., size 14×14×40) and the fourth RGB feature map (e.g., size 7×7×96).
[0040] S300: Input the fused feature information and multimodal image into the fresh weight prediction model to obtain the fresh weight of the lettuce to be detected as output by the fresh weight prediction model.
[0041] Among them, the fusion feature model is trained on the multimodal images of the samples based on the first preset convolutional neural network model; the freshness prediction model is trained on the second preset convolutional neural network model based on the fusion feature information of the samples, the multimodal images of the samples, and the labels of the freshness weight of the samples. The single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model. The freshness prediction model generates an attention feature map based on the fusion feature information and the multimodal images. Based on the attention feature map, a multimodal feature map is generated. The multimodal feature map is mapped to freshness weight. The attention feature map includes spatial attention feature map and channel attention feature map.
[0042] Spatial attention feature maps include RGB spatial attention feature maps and depth spatial attention feature maps. A freshness prediction model is used to generate spatial attention feature maps: multiple RGB feature maps are numbered based on the feature extraction order; the last RGB feature map is subjected to average pooling to obtain the first average feature; the last RGB feature map is subjected to max pooling to obtain the first maximum feature; the first maximum feature and the first average feature are concatenated, convolved, and activated to obtain the first weight; the RGB spatial attention feature map is obtained based on the first weight and the last RGB image. The last RGB image is obtained by convolutional feature extraction based on the previous RGB single-modal feature map. The previous RGB single-modal feature map is obtained by summing the previous RGB channel attention feature map and the previous RGB spatial attention feature map. The B-space attention feature map is the preceding feature map adjacent to the RGB-space attention feature map. Multiple depth feature maps are numbered according to the feature extraction order. Average pooling is performed on the last depth feature map to obtain the second average feature. Max pooling is then performed on the last depth feature map to obtain the second maximum feature. The second maximum feature and the second average feature are concatenated, convolved, and activated to obtain the second weight. The depth space attention feature map is obtained based on the second weight and the last depth image. The last depth image is obtained by convolutional feature extraction based on the preceding depth unimodal feature map. The preceding depth unimodal feature map is obtained by summing the preceding depth channel attention feature map and the preceding depth space attention feature map. The preceding depth space attention feature map is the preceding feature map adjacent to the depth space attention feature map.
[0043] The formula for calculating the spatial attention feature map is:
[0044]
[0045] in, As the first weight, As the second weight, SF R For RGB spatial attention feature maps, SF D This is a deep spatial attention feature map. For the final RGB image, For the final depth image, This is the previous RGB single-modal feature map. This is the previous depthwise unimodal feature map. Conv represents the convolution operation, Conv3 is a convolutional layer with a 3×3 kernel, Cat is the concatenation process, MaxPool is the max pooling process, and AvgPool is the average pooling process. This is the last RGB feature map. This is the last depth feature map.
[0046] The channel attention feature map includes the RGB channel attention feature map and the depth channel attention feature map. The freshness prediction model is used to generate the channel attention feature map: coordinate attention is extracted from the last RGB image to obtain the initial RGB channel attention feature map, and coordinate attention is extracted from the last depth image to obtain the initial depth channel attention feature map; based on the fusion result of the initial RGB channel attention feature map and the initial depth channel attention feature map, the RGB channel attention feature map and the depth channel attention feature map are obtained.
[0047] The formula for calculating the channel attention feature map is:
[0048]
[0049] Where CA stands for coordinate attention extraction, and MIF stands for fusion (multimodal fusion attention mechanism). For the final depth image, For the final RGB image, EF D For the initial depth channel attention feature map, EF R For the initial RGB channel attention feature map, CF D Deep Channel Attention Feature Map, CF R This is the attention feature map for the RGB channels.
[0050] The freshness prediction model is used to generate multimodal feature maps: An RGB unimodal feature map is obtained by summing the RGB channel attention feature map and the RGB spatial attention feature map; the RGB unimodal feature map is then convolved and subjected to global average pooling to obtain the target RGB feature map; a depth unimodal feature map is obtained by summing the depth channel attention feature map and the depth spatial attention feature map; the depth unimodal feature map is then convolved and subjected to global average pooling to obtain the target depth feature map; finally, the target RGB feature map and the target depth feature map are concatenated and convolved to obtain the multimodal feature map.
[0051] The formula for calculating multimodal feature maps is:
[0052]
[0053] in, For RGB single-modal feature maps, SF R For RGB spatial attention feature maps, CF R This is the attention feature map for the RGB channels. For deep single-modal feature maps, SF D For deep spatial attention feature maps, CF DFor deep channel attention feature maps, CBR is a combination of 3×3 convolution kernels, normalization, and ReLU layers; FCBR is a combination of 1×1 convolution kernels, normalization, and ReLU layers; Cat is for concatenation; F is for multimodal feature maps; and GAP is for global average pooling.
[0054] The first weight is obtained from the first RGB feature map. The first RGB spatial attention feature map is obtained from the first weight and the first RGB image (obtained by convolutional feature extraction of the RGB image input to the fresh weight prediction model for the first time). The second weight is obtained from the first depth feature map. The first depth spatial attention feature map is obtained from the second weight and the first depth image (obtained by convolutional feature extraction of the depth image input to the fresh weight prediction model for the first time).
[0055] Coordinate attention is extracted from the first RGB image to obtain the first initial RGB channel attention feature map. Coordinate attention is extracted from the first depth image to obtain the first initial depth channel attention feature map. The first initial RGB channel attention feature map and the first initial depth channel attention feature map are fused using a multimodal fusion attention mechanism to obtain the first RGB channel attention feature map and the first depth channel attention feature map.
[0056] The first RGB unimodal feature map is obtained by summing the first RGB channel attention feature map and the first RGB spatial attention feature map. Convolutional feature extraction is then performed on the first RGB unimodal feature map to obtain the second RGB image. Similarly, the first depth unimodal feature map is obtained by summing the first depth channel attention feature map and the first depth spatial attention feature map. Convolutional feature extraction is then performed on the first depth unimodal feature map to obtain the second depth image. This process is repeated until the previous RGB unimodal feature map is obtained. Compared to the previous depth single-modal feature map Convolutional feature extraction is performed on the previous RGB single-modal feature map to obtain the final depth image. Convolutional feature extraction is performed on the previous depthwise unimodal feature map to obtain the final RGB image.
[0057] Based on the above steps, a multimodal feature map is obtained from the last depth image, the last RGB image, the last RGB feature map, and the last depth feature map. The freshness weight prediction model further maps the multimodal feature map to freshness weight.
[0058] The fresh weight determination method for lettuce provided in this invention involves acquiring a multimodal image of the lettuce to be detected, including an RGB image and a depth image; inputting the preprocessed multimodal image into at least one fusion feature model to obtain fusion feature information output by the fusion feature model; inputting the fusion feature information and the multimodal image into a fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model; wherein, the fusion feature model is trained based on the sample multimodal image on the basis of a first preset convolutional neural network model; the fresh weight prediction model is trained based on the sample fusion feature information, the sample multimodal image, and the label of the sample fresh weight on the basis of a second preset convolutional neural network model; the single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model; the fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal image; generates a multimodal feature map based on the attention feature map; and maps the multimodal feature map to fresh weight; the attention feature map includes a spatial attention feature map and a channel attention feature map. This invention achieves accurate fusion of RGB and depth images of lettuce by integrating a feature model and a fresh weight prediction model, which improves the accuracy of lettuce fresh weight estimation. The fresh weight prediction model automatically obtains the fresh weight using artificial intelligence technology, resulting in a high degree of automation. This effectively reduces manual intervention in greenhouse lettuce cultivation and management, lowers application costs and complexity, and significantly improves the accuracy and real-time performance of greenhouse lettuce fresh weight estimation.
[0059] Based on the above embodiments, the encoder of the fusion feature model includes an RGB feature encoder and a deep feature encoder, and the freshness prediction model is obtained based on the following steps:
[0060] S400: Uses the RGB feature encoder as the RGB single-modal feature extractor of the second preset convolutional neural network model, and uses the deep feature encoder as the deep single-modal feature extractor of the second preset convolutional neural network model.
[0061] S500: A second pre-defined convolutional neural network model is constructed based on the input layer, RGB single-modal feature extractor, deep single-modal feature extractor, multimodal feature fusion module and output layer.
[0062] S600: Using the freshness weight of the sample as the third label, the third training sample is obtained based on the third label, the multimodal image of the sample, and the fusion feature information of the matching sample. The fusion feature information of the matching sample is the fusion feature information of the sample that matches the multimodal image of the sample.
[0063] S700: The second preset convolutional neural network model is trained based on the third training sample to obtain the freshness weight prediction model.
[0064] like Figure 3As shown, a second pre-defined convolutional neural network model is constructed. This model includes one input layer, one multimodal feature extractor, and one output layer. The multimodal feature extractor comprises two parallel single-modal feature extractors (a deep single-modal feature extractor and an RGB single-modal feature extractor) and a multimodal feature fusion module. The deep single-modal feature extractor uses the deep feature encoder part of the RGB fusion feature model. The RGB single-modal feature extractor uses the RGB feature encoder part of the deep fusion feature model.
[0065] When training the second preset convolutional neural network model, the sample freshness weight is used as the third label. Based on the third label, the sample fusion feature information and the corresponding multimodal images of the samples are labeled, resulting in a third training sample with labels. The second preset convolutional neural network model is trained based on the third training sample until the output error of the second preset convolutional neural network model is less than a third preset threshold, at which point training ends, and the freshness weight prediction model is obtained.
[0066] This invention achieves accurate fusion of RGB feature maps and depth feature maps through a fresh weight prediction model, and predicts fresh weight based on the fusion result, thereby improving the accuracy and efficiency of fresh weight prediction.
[0067] Based on the above embodiments, the fusion feature model includes an RGB fusion feature model and a deep fusion feature model, the first preset convolutional neural network model includes a preset RGB model and a preset deep model, and the fusion feature model is obtained based on the following steps:
[0068] S800: Using the sample depth image as the first label, the first training sample is obtained based on the first label and the sample RGB image. The preset RGB model is trained based on the first training sample to obtain the RGB fusion feature model.
[0069] S900: Using the sample RGB image as the second label, the second training sample is obtained based on the second label and the sample depth image. The preset depth model is trained based on the second training sample to obtain the deep fusion feature model.
[0070] Both the RGB fusion feature model and the deep fusion feature model employ a Unet-like encoder-decoder architecture. For example, the default RGB model uses a Unet structure with a Squeeze-and-Excitation ResNet-18 (SeResnet18) encoder. The default deep model uses a Unet structure with MobileNetV3_Large as the encoder.
[0071] like Figure 3As shown, an RGB fusion feature model and a deep fusion feature model are constructed through bidirectional modeling training. When training the preset RGB model, the sample depth image is used as the first label. The sample RGB images are labeled according to the first label to obtain the first training sample carrying the label. The preset RGB model is trained based on the first training sample until the output error of the preset RGB model is less than a first set threshold, at which point training ends, and the RGB fusion feature model is obtained.
[0072] When training the preset depth model, the sample RGB image is used as the second label. The sample depth image is labeled according to the second label to obtain the labeled second training sample. The preset depth model is trained according to the second training sample until the output error of the preset depth model is less than a second set threshold, then the training ends and the deep fusion feature model is obtained.
[0073] This invention achieves efficient feature extraction from RGB and depth images of lettuce by training a fusion feature model.
[0074] This invention also provides a device for determining the fresh weight of lettuce, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the lettuce fresh weight determination device provided by the present invention. It should be noted that the lettuce fresh weight determination device provided in this embodiment can execute the lettuce fresh weight determination method described in any of the above embodiments during actual operation; however, this embodiment will not elaborate further on this.
[0075] Reference Figure 4 An embodiment of the present invention provides a device for determining the fresh weight of lettuce, comprising:
[0076] The multimodal image acquisition module 401 is used to acquire multimodal images of the lettuce to be detected, including RGB images and depth images.
[0077] The fusion feature information determination module 402 is used to input the preprocessed multimodal image into at least one fusion feature model and obtain the fusion feature information output by the fusion feature model.
[0078] The fresh weight prediction module 403 is used to input fused feature information and multimodal images into the fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model. The fused feature model is trained on the sample multimodal images based on the first preset convolutional neural network model. The fresh weight prediction model is trained on the sample fused feature information, sample multimodal images and sample fresh weight labels based on the second preset convolutional neural network model. The single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fused feature model. The fresh weight prediction model generates an attention feature map based on the fused feature information and multimodal images, generates a multimodal feature map based on the attention feature map, and maps the multimodal feature map to fresh weight. The attention feature map includes a spatial attention feature map and a channel attention feature map.
[0079] The fresh weight determination device for lettuce provided in this embodiment of the invention acquires a multimodal image of the lettuce to be detected, the multimodal image including an RGB image and a depth image; inputs the preprocessed multimodal image into at least one fusion feature model to obtain fusion feature information output by the fusion feature model; inputs the fusion feature information and the multimodal image into a fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model; wherein, the fusion feature model is trained based on the sample multimodal image on the basis of a first preset convolutional neural network model; the fresh weight prediction model is trained based on the sample fusion feature information, the sample multimodal image and the label of the sample fresh weight on the basis of a second preset convolutional neural network model, the single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model, the fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal image, generates a multimodal feature map based on the attention feature map, and maps the multimodal feature map to fresh weight, the attention feature map including a spatial attention feature map and a channel attention feature map. This invention achieves accurate fusion of RGB and depth images of lettuce by integrating a feature model and a fresh weight prediction model, which improves the accuracy of lettuce fresh weight estimation. The fresh weight prediction model automatically obtains the fresh weight using artificial intelligence technology, resulting in a high degree of automation. This effectively reduces manual intervention in greenhouse lettuce cultivation and management, lowers application costs and complexity, and significantly improves the accuracy and real-time performance of greenhouse lettuce fresh weight estimation.
[0080] In one embodiment, the encoder of the fusion feature model includes an RGB feature encoder and a deep feature encoder. The freshness prediction module 403 is used to: use the RGB feature encoder as the RGB single-modal feature extractor of the second preset convolutional neural network model, and use the deep feature encoder as the deep single-modal feature extractor of the second preset convolutional neural network model; construct the second preset convolutional neural network model based on the input layer, the RGB single-modal feature extractor, the deep single-modal feature extractor, the multimodal feature fusion module, and the output layer; use the sample freshness as the third label, and obtain the third training sample based on the third label, the sample multimodal image, and the matching sample fusion feature information, where the matching sample fusion feature information is the sample fusion feature information that matches the sample multimodal image; and train the second preset convolutional neural network model based on the third training sample to obtain the freshness prediction model.
[0081] In one embodiment, the sample multimodal image includes a sample depth image and a sample RGB image, the fusion feature model includes an RGB fusion feature model and a depth fusion feature model, the first preset convolutional neural network model includes a preset RGB model and a preset depth model, and the fusion feature information determination module 402 is used to: use the sample depth image as a first label, obtain a first training sample based on the first label and the sample RGB image, train the preset RGB model based on the first training sample to obtain an RGB fusion feature model; use the sample RGB image as a second label, obtain a second training sample based on the second label and the sample depth image, train the preset depth model based on the second training sample to obtain a depth fusion feature model.
[0082] In one embodiment, the fused feature information includes multiple RGB feature maps and multiple depth feature maps. The fused feature information determination module 402 is used to: perform multiple depth feature extractions on the RGB image sequentially using a depth feature encoder based on the RGB fusion feature model, and obtain depth feature maps based on the results of each depth feature extraction to obtain multiple depth feature maps; and perform multiple RGB feature extractions on the depth image sequentially using an RGB feature extractor based on the depth fusion feature model, and obtain RGB feature maps based on the results of each RGB feature extraction to obtain multiple RGB feature maps.
[0083] In one embodiment, the spatial attention feature map includes an RGB spatial attention feature map and a depth spatial attention feature map. The freshness prediction module 403 is used to: number multiple RGB feature maps based on the feature extraction order; perform average pooling on the last RGB feature map to obtain a first average feature; perform max pooling on the last RGB feature map to obtain a first maximum feature; concatenate, convolve, and activate the first maximum feature and the first average feature to obtain a first weight; and obtain an RGB spatial attention feature map based on the first weight and the last RGB image. The last RGB image is obtained by convolutional feature extraction based on the previous RGB single-modal feature map. The previous RGB single-modal feature map is obtained by summing the previous RGB channel attention feature map and the previous RGB spatial attention feature map. The GB spatial attention feature map is the preceding feature map adjacent to the RGB spatial attention feature map. Multiple depth feature maps are numbered according to the feature extraction order. Average pooling is performed on the last depth feature map to obtain the second average feature. Max pooling is then performed on the last depth feature map to obtain the second maximum feature. The second maximum feature and the second average feature are concatenated, convolved, and activated to obtain the second weight. The depth spatial attention feature map is obtained based on the second weight and the last depth image. The last depth image is obtained by convolutional feature extraction based on the preceding depth unimodal feature map. The preceding depth unimodal feature map is obtained by summing the preceding depth channel attention feature map and the preceding depth spatial attention feature map. The preceding depth spatial attention feature map is the preceding feature map adjacent to the depth spatial attention feature map.
[0084] In one embodiment, the channel attention feature map includes an RGB channel attention feature map and a depth channel attention feature map. The freshness prediction module 403 is used to: extract coordinate attention from the last RGB image to obtain an initial RGB channel attention feature map; extract coordinate attention from the last depth image to obtain an initial depth channel attention feature map; and obtain an RGB channel attention feature map and a depth channel attention feature map based on the fusion result of the initial RGB channel attention feature map and the initial depth channel attention feature map.
[0085] In one embodiment, the freshness prediction module 403 is used to: obtain an RGB unimodal feature map by summing the RGB channel attention feature map and the RGB spatial attention feature map; perform convolution and global average pooling on the RGB unimodal feature map to obtain a target RGB feature map; obtain a depth unimodal feature map by summing the depth channel attention feature map and the depth spatial attention feature map; perform convolution and global average pooling on the depth unimodal feature map to obtain a target depth feature map; and concatenate and convolve the target RGB feature map and the target depth feature map to obtain a multimodal feature map.
[0086] In one embodiment, the feature information fusion determination module 402 is used to: crop the multimodal image to obtain a cropped image, the cropped image being centered on the lettuce to be detected and including the complete lettuce to be detected; and to perform standardization processing on the cropped image based on the cropped image, the minimum pixel value of the lettuce region of the cropped image, and the maximum pixel value of the lettuce region of the cropped image to obtain a preprocessed multimodal image, the lettuce region being the image region including the lettuce to be detected.
[0087] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a method for determining the fresh weight of lettuce. This method includes: acquiring a multimodal image of the lettuce to be detected, the multimodal image including an RGB image and a depth image; inputting the preprocessed multimodal image into at least one fusion feature model to obtain fusion feature information output by the fusion feature model; inputting the fusion feature information and the multimodal image into a fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model; wherein the fusion feature model is trained based on a first preset convolutional neural network model using sample multimodal images; the fresh weight prediction model is trained based on a second preset convolutional neural network model using sample fusion feature information, sample multimodal images, and labels of sample fresh weight; the single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model; the fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal image; generates a multimodal feature map based on the attention feature map; and maps the multimodal feature map to fresh weight; the attention feature map includes a spatial attention feature map and a channel attention feature map.
[0088] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for determining the fresh weight of lettuce, characterized in that, include: Acquire a multimodal image of the lettuce to be detected, wherein the multimodal image includes an RGB image and a depth image; The preprocessed multimodal image is input into at least one fusion feature model to obtain the fusion feature information output by the fusion feature model; The fused feature information and the multimodal image are input into the fresh weight prediction model to obtain the fresh weight of the lettuce to be detected, as output by the fresh weight prediction model. The fusion feature model includes an RGB fusion feature model and a deep fusion feature model, which is obtained based on the following steps: Using the sample depth image as the first label, a first training sample is obtained based on the first label and the sample RGB image. The preset RGB model is then trained based on the first training sample to obtain the RGB fusion feature model. Using the sample RGB image as the second label, a second training sample is obtained based on the second label and the sample depth image. The preset depth model is trained based on the second training sample to obtain the depth fusion feature model. The fresh weight prediction model is trained based on the sample fusion feature information, sample multimodal images, and sample fresh weight labels, on the basis of the second preset convolutional neural network model. The single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model. The fresh weight prediction model generates an attention feature map based on the fusion feature information and the multimodal image, generates a multimodal feature map based on the attention feature map, and maps the multimodal feature map to the fresh weight. The attention feature map includes a spatial attention feature map and a channel attention feature map. The fused feature information includes multiple RGB feature maps and multiple depth feature maps, and the fused feature model is used to generate the fused feature information. The deep feature encoder based on the RGB fusion feature model performs multiple deep feature extractions on the RGB image in sequence, and obtains the deep feature map based on the result of each deep feature extraction, so as to obtain multiple deep feature maps; The RGB feature extractor based on the deep fusion feature model performs multiple RGB feature extractions on the depth image sequentially, and obtains the RGB feature map based on the result of each RGB feature extraction, so as to obtain multiple RGB feature maps.
2. The method for determining the fresh weight of lettuce according to claim 1, characterized in that, The encoder of the fusion feature model includes an RGB feature encoder and a deep feature encoder, and the freshness weight prediction model is obtained based on the following steps: The RGB feature encoder is used as the RGB single-modal feature extractor of the second preset convolutional neural network model, and the deep feature encoder is used as the deep single-modal feature extractor of the second preset convolutional neural network model. The second preset convolutional neural network model is constructed based on the input layer, the RGB single-modal feature extractor, the deep single-modal feature extractor, the multimodal feature fusion module, and the output layer; Using the fresh weight of the sample as the third label, a third training sample is obtained based on the third label, the multimodal image of the sample, and the matching sample fusion feature information. The matching sample fusion feature information is the sample fusion feature information that matches the multimodal image of the sample. The second preset convolutional neural network model is trained based on the third training sample to obtain the fresh weight prediction model.
3. The method for determining the fresh weight of lettuce according to claim 1, characterized in that, The spatial attention feature map includes an RGB spatial attention feature map and a depth spatial attention feature map, and the freshness-weight prediction model is used to generate the spatial attention feature map. The RGB feature maps are numbered according to the feature extraction order. The last RGB feature map is subjected to average pooling to obtain the first average feature. The last RGB feature map is subjected to max pooling to obtain the first maximum feature. The first maximum feature and the first average feature are concatenated, convolved, and activated to obtain the first weight. The RGB spatial attention feature map is obtained based on the first weight and the last RGB image. The last RGB image is obtained by convolutional feature extraction based on the previous RGB single-modal feature map. The previous RGB single-modal feature map is obtained by summing the previous RGB channel attention feature map and the previous RGB spatial attention feature map. The previous RGB spatial attention feature map is the feature map adjacent to the previous RGB spatial attention feature map. The multiple depth feature maps are numbered according to the feature extraction order. The last depth feature map is subjected to average pooling to obtain the second average feature. The last depth feature map is subjected to max pooling to obtain the second maximum feature. The second maximum feature and the second average feature are concatenated, convolved, and activated to obtain the second weight. The depth spatial attention feature map is obtained based on the second weight and the last depth image. The last depth image is obtained by convolutional feature extraction based on the previous depth unimodal feature map. The previous depth unimodal feature map is obtained by summing the previous depth channel attention feature map and the previous depth spatial attention feature map. The previous depth spatial attention feature map is the feature map adjacent to the previous depth spatial attention feature map.
4. The method for determining the fresh weight of lettuce according to claim 3, characterized in that, The channel attention feature map includes an RGB channel attention feature map and a depth channel attention feature map, and the freshness-weight prediction model is used to generate the channel attention feature map. Coordinate attention is extracted from the last RGB image to obtain an initial RGB channel attention feature map, and coordinate attention is extracted from the last depth image to obtain an initial depth channel attention feature map. Based on the fusion result of the initial RGB channel attention feature map and the initial depth channel attention feature map, the RGB channel attention feature map and the depth channel attention feature map are obtained.
5. The method for determining the fresh weight of lettuce according to claim 4, characterized in that, The fresh weight prediction model is used to generate the multimodal feature map: Based on the sum of the RGB channel attention feature map and the RGB spatial attention feature map, an RGB unimodal feature map is obtained. The RGB unimodal feature map is then subjected to convolution and global average pooling to obtain the target RGB feature map. Based on the sum of the deep channel attention feature map and the deep spatial attention feature map, a deep unimodal feature map is obtained. The deep unimodal feature map is then subjected to convolution and global average pooling to obtain the target depth feature map. The target RGB feature map and the target depth feature map are concatenated and convolved to obtain the multimodal feature map.
6. The method for determining the fresh weight of lettuce according to claim 1, characterized in that, The preprocessed multimodal image is obtained based on the following steps: The multimodal image is cropped to obtain a cropped image, which includes the complete lettuce to be detected, with the lettuce to be detected as the center. Based on the cropped image, the minimum pixel value of the lettuce region in the cropped image, and the maximum pixel value of the lettuce region in the cropped image, the cropped image is standardized to obtain the preprocessed multimodal image, wherein the lettuce region is the image region including the lettuce to be detected.
7. A device for determining the fresh weight of lettuce, characterized in that, include: A multimodal image acquisition module is used to acquire a multimodal image of the lettuce to be detected, wherein the multimodal image includes an RGB image and a depth image; The fusion feature information determination module is used to input the preprocessed multimodal image into at least one fusion feature model and obtain the fusion feature information output by the fusion feature model; The fresh weight prediction module is used to input the fused feature information and the multimodal image into the fresh weight prediction model to obtain the fresh weight of the lettuce to be detected output by the fresh weight prediction model. The fusion feature model includes an RGB fusion feature model and a deep fusion feature model. The fusion feature model is obtained based on the following steps: using the sample depth image as the first label, obtaining a first training sample based on the first label and the sample RGB image, and training a preset RGB model based on the first training sample to obtain the RGB fusion feature model; using the sample RGB image as the second label, obtaining a second training sample based on the second label and the sample depth image, and training a preset depth model based on the second training sample to obtain the deep fusion feature model; the freshness prediction model is obtained by training a second preset convolutional neural network model based on sample fusion feature information, sample multimodal images, and sample freshness labels. The single-modal feature extractor of the second preset convolutional neural network model is constructed based on the encoder of the fusion feature model. The freshness prediction model generates an attention feature map based on the fusion feature information and the multimodal image, generates a multimodal feature map based on the attention feature map, and maps the multimodal feature map to the freshness. The attention feature map includes a spatial attention feature map and a channel attention feature map. The fused feature information includes multiple RGB feature maps and multiple depth feature maps. The fused feature information determination module is further configured to perform multiple depth feature extractions on the RGB image sequentially using a depth feature encoder based on the RGB fusion feature model, and obtain the depth feature map based on the result of each depth feature extraction, so as to obtain multiple depth feature maps; and to perform multiple RGB feature extractions on the depth image sequentially using an RGB feature extractor based on the depth fusion feature model, and obtain the RGB feature map based on the result of each RGB feature extraction, so as to obtain multiple RGB feature maps.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for determining the fresh weight of lettuce as described in any one of claims 1 to 6.