Method, device, equipment and product for monitoring growth parameters of greenhouse tomato plants

Abstract

The present application relates to the field of agricultural technologies, and provides a method, device, equipment and product for monitoring the growth parameters of greenhouse tomato plants. The method includes: preprocessing the initial multi-modal data of tomato plants from multiple perspectives; performing fusion modeling on the obtained target multi-modal data to obtain three-dimensional reconstruction data of tomato plants; identifying key growth parts of tomato plants based on the three-dimensional reconstruction data, and obtaining target growth parameters of the key growth parts; and obtaining future predicted values of the target growth parameters based on the current time-series data of the target growth parameters. The present application increases the data collection dimension, and through multi-perspective collection, it avoids the influence of mutual overlap and occlusion between plants on the data. After three-dimensional reconstruction based on these data, it can accurately identify the key parts of the plants and obtain their target growth parameters, and then predict their future values through the current time-series data of the target growth parameters, realizing dynamic monitoring of the target growth parameters.

Description

Technical Field

[0001] This application relates to the field of agricultural technologies, and particularly to a method, device, equipment and product for monitoring the growth parameters of greenhouse tomato plants. Background Art

[0002] With the rapid development of modern agriculture, facility horticulture has become an important way to improve the comprehensive agricultural production capacity. As a representative crop of facility horticulture, the yield and quality of greenhouse tomatoes are directly related to the process of agricultural industrialization. Accurately monitoring the growth status of tomato plants is the key to realizing the digital and intelligent management of tomato production.

[0003] However, the current monitoring of the growth of greenhouse tomato plants mainly relies on manual monitoring and semi-automatic monitoring. Manual monitoring visually observes and simply measures the growth status of plants based on experience, with deficiencies such as low monitoring efficiency and accuracy; although semi-automatic monitoring introduces machine vision analysis to extract some growth parameters of plants, most of them are based on two-dimensional images, with a single feature dimension, and due to the fact that plants often have varying degrees of overlap and occlusion during the growth process, it is difficult to accurately obtain the complete growth parameters of plants from two-dimensional images, thus affecting the monitoring accuracy; furthermore, whether it is manual monitoring or semi-automatic monitoring, it can only monitor the current growth parameters of plants based on the current visual observation or two-dimensional images, and cannot dynamically monitor the continuous changes in the growth parameters of plants during the growth process.

[0004] In summary, the current method for monitoring the growth of greenhouse tomato plants is based on manual monitoring and semi-automatic monitoring, resulting in low monitoring efficiency and accuracy, and being unable to dynamically monitor the continuous changes in the growth parameters of plants during the growth process. Summary of the Invention

[0005] Embodiments of this application provide a method, device, equipment and product for monitoring the growth parameters of greenhouse tomato plants, so as to solve the technical problem that the current method for monitoring the growth of greenhouse tomato plants is based on manual monitoring and semi-automatic monitoring, resulting in low monitoring efficiency and accuracy, and being unable to dynamically monitor the continuous changes in the growth parameters of plants during the growth process.

[0006] In a first aspect, embodiments of this application provide a method for monitoring the growth parameters of greenhouse tomato plants, including:

[0007] Obtaining initial multi-modal data of tomato plants collected by a collection trolley in a target greenhouse from multiple perspectives; the initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data;

[0008] Preprocessing the initial multi-modal data to obtain target multi-modal data;

[0009] Fuse and model the target multi-modal data to obtain the three-dimensional reconstruction data of the tomato plant;

[0010] Identify the key growth parts of the tomato plant based on the three-dimensional reconstruction data;

[0011] Obtain the target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts;

[0012] Obtain the future predicted values of the target growth parameters based on the current time series data of the target growth parameters.

[0013] In one embodiment, the initial multi-modal data from multiple perspectives is collected based on the following method:

[0014] Based on the positions of multiple tomato plants in the target greenhouse, the already collected positions adjacent to the multiple tomato plants, the position of the collection trolley, and the speed of the collection trolley, construct the target path function of the collection trolley; the speed includes linear speed and angular speed;

[0015] Adjust the linear speed and angular speed of the collection trolley until the value of the target path function is minimized, and determine the linear speed and angular speed of the collection trolley at this time as the optimal linear speed and optimal angular speed;

[0016] Obtain the optimal heading angle of the collection trolley based on the optimal angular speed;

[0017] Based on the optimal linear speed and the optimal heading angle, control the movement of the collection trolley and collect the multi-modal data of each tomato plant in the target greenhouse to obtain the initial multi-modal data from multiple perspectives.

[0018] In one embodiment, the fusing and modeling of the target multi-modal data to obtain the three-dimensional reconstruction data of the tomato plant includes:

[0019] Input the target multi-modal data into a pre-trained semantic segmentation network to obtain the target multi-modal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network; the target multi-modal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data, and target three-dimensional point cloud semantic segmentation data;

[0020] Calculate the attention weights of the target multi-modal semantic segmentation data of different parts in the feature dimension and the spatial dimension to obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data;

[0021] Based on the first feature attention weight and the first spatial attention weight, obtain the texture feature of the target RGB image data;

[0022] Based on the second feature attention weight and the second spatial attention weight, obtain the depth feature of the target depth image data;

[0023] Based on the third feature attention weight and the third spatial attention weight, obtain the normal vector feature of the target 3D point cloud data;

[0024] Calculate the attention weights between every two of the texture feature, the depth feature, and the normal vector feature, and obtain the first correlation weight between the texture feature and the depth feature, the second correlation weight between the texture feature and the normal vector feature, and the third correlation weight between the depth feature and the normal vector feature;

[0025] Based on the first correlation weight and the second correlation weight, obtain the first fusion feature with the texture feature as the central feature;

[0026] Based on the first correlation weight and the third correlation weight, obtain the second fusion feature with the depth feature as the central feature;

[0027] Based on the second correlation weight and the third correlation weight, obtain the third fusion feature with the normal vector feature as the central feature;

[0028] Perform weighted summation on the first fusion feature, the second fusion feature, and the third fusion feature to obtain the 3D reconstruction data of the tomato plant.

[0029] In one embodiment, the identifying the key growth parts of the tomato plant based on the 3D reconstruction data includes:

[0030] Input the 3D reconstruction data into the Swin Transformer network to obtain the 3D reconstruction features in multiple stages of the Swin Transformer network;

[0031] Aggregate the 3D reconstruction features of adjacent stages to obtain multiple first aggregation features;

[0032] Group the multiple first aggregation features to obtain multiple groups of first aggregation features;

[0033] Calculate the attention weights of each group of first aggregation features, and aggregate the multiple groups of first aggregation features based on the attention weights to obtain the second aggregation feature;

[0034] Identify the key growth parts of the tomato plant based on the second aggregation feature.

[0035] In one embodiment, obtaining the target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts includes:

[0036] Input the three-dimensional reconstruction data corresponding to the key growth parts into the target growth parameter recognition model to obtain the target growth parameters output by the target growth parameter recognition model;

[0037] The key growth parts include the flowers of the tomato plant, the leaves of the tomato plant, the stems of the tomato plant, and the vertices of the tomato plant, and the target growth parameters include the growth period category of the flowers, the length of the leaves, the stem diameter of the stems, and the height of the vertices;

[0038] The target growth parameter recognition model is trained by using the historical three-dimensional reconstruction data of the key growth parts and their corresponding target growth parameter labels to train a basic model that matches the recognition of the target growth parameters.

[0039] In one embodiment, obtaining the future prediction value of the target growth parameter based on the current time series data of the target growth parameter includes:

[0040] Generate the lag time series data and the moving average time series data of the target growth parameter based on the current time series data of the target growth parameter;

[0041] Concatenate the current time series data, the lag time series data, and the moving average time series data to obtain concatenated time series data;

[0042] Perform convolution and pooling on the concatenated time series data to obtain multiple local features of the target growth parameter in the spatio-temporal dimension;

[0043] Capture the long-term dependence relationship between each local feature and the local features in the front and back time series to obtain multiple long-term dependence local features of the target growth parameter in the spatio-temporal dimension;

[0044] Calculate the attention weights between the multiple long-term dependence local features;

[0045] Based on the attention weights, calculate the hidden state representation of the target growth parameter in the spatio-temporal dimension with each long-term dependence local feature as the center to obtain multiple hidden state representations;

[0046] Map the multiple hidden state representations to the output space to obtain the future prediction value of the target growth parameter.

[0047] In one embodiment, the preprocessing of the initial multimodal data to obtain target multimodal data includes:

[0048] Performing exposure correction, image enhancement, distortion correction, and denoising on the initial RGB image data in sequence to obtain the RGB image data to be processed;

[0049] Performing invalid value filling and denoising on the initial depth image data in sequence to obtain the depth image data to be processed;

[0050] Performing outlier removal, downsampling, and normal vector estimation on the initial 3D point cloud data in sequence to obtain the 3D point cloud data to be processed;

[0051] Aligning the RGB image data to be processed, the depth image data to be processed, and the 3D point cloud data to be processed to obtain target RGB image data, target depth image data, and target 3D point cloud data.

[0052] In a second aspect, an embodiment of the present application provides a greenhouse tomato plant growth parameter monitoring device, including:

[0053] An initial multimodal data acquisition module, configured to: acquire the initial multimodal data of a tomato plant from multiple perspectives collected by a collection trolley in a target greenhouse; the initial multimodal data includes initial RGB image data, initial depth image data, and initial 3D point cloud data;

[0054] A target multimodal data acquisition module, configured to: preprocess the initial multimodal data to obtain target multimodal data;

[0055] A 3D reconstruction data acquisition module, configured to perform fusion modeling on the target multimodal data to obtain the 3D reconstruction data of the tomato plant;

[0056] A key growth part identification module, configured to identify the key growth parts of the tomato plant based on the 3D reconstruction data;

[0057] A target growth parameter acquisition module, configured to obtain the target growth parameters of the key growth parts based on the 3D reconstruction data corresponding to the key growth parts;

[0058] A target growth parameter prediction module, configured to obtain future predicted values of the target growth parameters based on the current time series data of the target growth parameters.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a computer program, and when the processor executes the program, the steps of the greenhouse tomato plant growth parameter monitoring method described in the first aspect are implemented.

[0060] Fourthly, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the greenhouse tomato plant growth parameter monitoring method described in the first aspect.

[0061] The greenhouse tomato plant growth parameter monitoring method, device, equipment and product provided by the present application obtain the initial multi-modal data of tomato plants at multiple perspectives collected by a collection trolley in a target greenhouse. The initial multi-modal data includes initial RGB image data, initial depth image data and initial three-dimensional point cloud data. The initial multi-modal data is preprocessed to obtain target multi-modal data, the target multi-modal data is fused and modeled to obtain three-dimensional reconstruction data of the tomato plants, the key growth parts of the tomato plants are identified based on the three-dimensional reconstruction data, the target growth parameters of the key growth parts are obtained based on the three-dimensional reconstruction data corresponding to the key growth parts, and the future predicted values of the target growth parameters are obtained based on the current time series data of the target growth parameters. By obtaining the RGB image data, depth image data and three-dimensional point cloud data of tomato plants at multiple perspectives, on the one hand, the data collection dimension is increased, and on the other hand, through multi-perspective collection, the influence of mutual overlap and occlusion between plants on the integrity of data acquisition can be avoided. After three-dimensional reconstruction of the plants based on these data, the key parts of the plants can be accurately identified and their target growth parameters can be obtained, and by predicting the future values of the target growth parameters through the current time series data of the target growth parameters, the dynamic growth data of the target growth parameters can be obtained, so as to realize the dynamic monitoring of the target growth parameters of the plants. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0063] Figure 1 is one of the flow diagrams of the greenhouse tomato plant growth parameter monitoring method provided by the embodiment of the present application;

[0064] Figure 2 is the second flow diagram of the greenhouse tomato plant growth parameter monitoring method provided by the embodiment of the present application;

[0065] Figure 3 is the third flow diagram of the greenhouse tomato plant growth parameter monitoring method provided by the embodiment of the present application;

[0066] Figure 4 is the fourth flow diagram of the greenhouse tomato plant growth parameter monitoring method provided by the embodiment of the present application;

[0067] Figure 5 It is the fifth flow schematic diagram of the greenhouse tomato plant growth parameter monitoring method provided by the embodiments of the present application;

[0068] Figure 6 It is the structural schematic diagram of the greenhouse tomato plant growth parameter monitoring device provided by the embodiments of the present application;

[0069] Figure 7 It is the structural schematic diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0070] To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0071] It should be noted that in the description of the embodiments of the present application, the terms "include", "comprise" or any other variation thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, the element defined by the phrase "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element. The orientation or positional relationship indicated by terms such as "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of the present application. Unless otherwise clearly defined and limited, the terms "install", "connect", "couple" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0072] The terms "first", "second", etc. in this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and do not limit the number of objects. For example, the first object can be one or multiple. In addition, "and / or" means at least one of the connected objects, and the character " / ", generally indicates an "or" relationship between the associated objects before and after.

[0073] Figure 1 is one of the schematic flowcharts of the greenhouse tomato plant growth parameter monitoring method provided by the embodiments of this application. Refer to Figure 1 , the embodiments of this application provide a greenhouse tomato plant growth parameter monitoring method, which may include:

[0074] 101. Obtain the initial multi-modal data of the tomato plants at multiple perspectives collected by the acquisition trolley in the target greenhouse;

[0075] The initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data;

[0076] 102. Preprocess the initial multi-modal data to obtain target multi-modal data;

[0077] 103. Perform fusion modeling on the target multi-modal data to obtain three-dimensional reconstruction data of the tomato plants;

[0078] 104. Identify the key growth parts of the tomato plants based on the three-dimensional reconstruction data;

[0079] 105. Obtain the target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts;

[0080] 106. Obtain the future predicted values of the target growth parameters based on the current time series data of the target growth parameters.

[0081] In step 101, an RGB camera, a depth camera, and a LiDAR (Light Detection and Ranging) sensor are installed on the acquisition trolley. When the acquisition trolley moves in the target greenhouse, the RGB camera, the depth camera, and the LiDAR sensor respectively collect the initial RGB image data, the initial depth image data, and the initial three-dimensional point cloud data of the tomato plants at high frequencies.

[0082] Among them, the RGB camera has a resolution of 4096 pixels × 3000 pixels, a pixel size of 3.45 microns, supports full-resolution output of up to 60 frames per second, and can capture the detailed texture of tomato plants. A 12-mm fixed-focus lens is used, and the field of view angle is approximately 90 degrees to obtain a large single-frame coverage range.

[0083] The depth camera has a resolution of 1024 pixels × 1024 pixels, a depth measurement range of 0.25 m to 2.88 m, a system error of less than 11 mm at a distance of 0.5 m, and less than 18 mm at a distance of 2 m. The system error increases linearly with the increase of distance. The frame rate is 30 frames per second and is triggered synchronously with the RGB camera.

[0084] The LiDAR sensor uses 16-beam lasers, with a vertical angular resolution of 2 degrees, a horizontal field of view angle of 360 degrees, a vertical field of view angle of 30 degrees, an error of ±15 degrees, a ranging range of 100 m, an accuracy of ±3 cm, and emits 300,000 laser pulses per second, which can generate precise three-dimensional point clouds.

[0085] The per-second pulse output signal of the LiDAR sensor is respectively connected to the external trigger input ports of the RGB camera and the depth camera. Each time the LiDAR sensor rotates one week, a synchronization signal is output. The two cameras are simultaneously exposed and record the trigger timestamp, and the timestamp accuracy reaches the microsecond level to achieve synchronous triggering of the two cameras.

[0086] In step 102, after preprocessing the initial RGB image data, the initial depth image data, and the initial three-dimensional point cloud data, the target RGB image data, the target depth image data, and the target three-dimensional point cloud data are correspondingly obtained.

[0087] The greenhouse tomato plant growth parameter monitoring method provided in this embodiment acquires the initial multi-modal data of tomato plants at multiple perspectives collected by a collection trolley in a target greenhouse. The initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data. The initial multi-modal data is preprocessed to obtain target multi-modal data. The target multi-modal data is fused and modeled to obtain three-dimensional reconstruction data of the tomato plants. Based on the three-dimensional reconstruction data, the key growth parts of the tomato plants are identified. Based on the three-dimensional reconstruction data corresponding to the key growth parts, the target growth parameters of the key growth parts are obtained. Based on the current time series data of the target growth parameters, the future predicted values of the target growth parameters are obtained. In this embodiment, by acquiring the RGB image data, depth image data, and three-dimensional point cloud data of tomato plants at multiple perspectives, on the one hand, the data acquisition dimension is increased, and on the other hand, through multi-perspective acquisition, the influence of mutual overlap and occlusion between plants on the integrity of data acquisition can be avoided. After three-dimensional reconstruction of the plants based on these data, the key parts of the plants can be accurately identified and their target growth parameters can be obtained. And by predicting the future values through the current time series data of the target growth parameters, the dynamic growth data of the target growth parameters can be obtained, so as to realize the dynamic monitoring of the target growth parameters of the plants.

[0088] Figure 2 It is the second schematic flow chart of the greenhouse tomato plant growth parameter monitoring method provided by the embodiments of the present application. Refer to Figure 2 In one embodiment, the initial multi-modal data at multiple perspectives can be collected based on the following method:

[0089] 201. Based on the positions of multiple tomato plants in the target greenhouse, the already collected positions adjacent to the multiple tomato plants, the position of the collection trolley, and the speed of the collection trolley, construct the target path function of the collection trolley;

[0090] The speed includes linear speed and angular speed;

[0091] 202. Adjust the linear speed and angular speed of the collection trolley until the value of the target path function is the smallest, and determine the linear speed and angular speed of the collection trolley at this time as the optimal linear speed and the optimal angular speed;

[0092] 203. Based on the optimal angular speed, obtain the optimal heading angle of the collection trolley;

[0093] 204. Based on the optimal linear speed and the optimal heading angle, control the movement of the collection trolley and collect the multi-modal data of each tomato plant in the target greenhouse to obtain the initial multi-modal data at multiple perspectives.

[0094] In steps 201 to 202, obtaining the minimum value of the target path function of the collection trolley can be expressed as:

[0095] ;

[0096] Among them, is the position of the collection trolley, is the th position of the th tomato plant among the th tomato plants, is the index set of the serial numbers of the already collected positions adjacent to the th tomato plant, and is the balance factor, is the derivative with respect to time, is the derivative with respect to time, is the heading angle of the collection trolley, is the derivative with respect to time, is the angular velocity of the collection trolley, is the set of the spatial range where the collection trolley can travel in the target greenhouse.

[0097] Based on the above formulas, the optimal linear velocity and the optimal angular velocity can be obtained.

[0098] In step 203, substituting into the formula based on , the optimal heading angle can be obtained.

[0099] In step 204, based on and , control the movement of the collection trolley and collect the multi-modal data of each tomato plant in the target greenhouse. It should be noted that during the movement of the collection trolley, and will also change in real time, enabling the collection trolley to always collect the data of tomato plants based on the current optimal linear velocity and heading angle, and obtaining the initial multi-modal data from multiple perspectives.

[0100] In this embodiment, by performing path planning on the driving route of the collection trolley, not only can the data collection of the collection trolley cover each tomato plant, but also by adding the distance between the position of the collection trolley and the already collected data positions adjacent to each tomato plant as a penalty term in the target path function, the collection trolley is encouraged to collect multiple types of initial multi-modal data from multiple perspectives near each plant, improving data diversity.

[0101] Figure 3 It is the third flowchart diagram of the greenhouse tomato plant growth parameter monitoring method provided by the embodiments of the present application. Refer to Figure 3 , in one embodiment, fusing and modeling the target multi-modal data to obtain the three-dimensional reconstruction data of the tomato plant may include:

[0102] 301. Input the target multi-modal data into a pre-trained semantic segmentation network to obtain the target multi-modal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network;

[0103] The target multi-modal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data, and target three-dimensional point cloud semantic segmentation data;

[0104] 302. Calculate the attention weights of the target multi-modal semantic segmentation data of different parts in the feature dimension and the spatial dimension to obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data;

[0105] 303. Based on the first feature attention weight and the first spatial attention weight, obtain the texture feature of the target RGB image data;

[0106] 304. Based on the second feature attention weight and the second spatial attention weight, obtain the depth feature of the target depth image data;

[0107] 305. Based on the third feature attention weight and the third spatial attention weight, obtain the normal vector feature of the target three-dimensional point cloud data;

[0108] 306. Calculate the attention weights between every two features among the texture feature, the depth feature, and the normal vector feature to obtain the first correlation weight between the texture feature and the depth feature, the second correlation weight between the texture feature and the normal vector feature, and the third correlation weight between the depth feature and the normal vector feature;

[0109] 307. Based on the first correlation weight and the second correlation weight, obtain the first fusion feature with the texture feature as the central feature;

[0110] 308. Based on the first correlation weight and the third correlation weight, obtain the second fusion feature with the depth feature as the central feature;

[0111] 309. Based on the second correlation weight and the third correlation weight, obtain the third fusion feature with the normal vector feature as the central feature;

[0112] 310. Weightedly sum up the first fusion feature, the second fusion feature, and the third fusion feature to obtain the three-dimensional reconstruction data of the tomato plant.

[0113] In step 301, different parts include the leaves, stems, flowers, etc. of the tomato plant. After being processed by the semantic segmentation network, the target RGB image semantic segmentation data, the target depth image semantic segmentation data, and the target three-dimensional point cloud semantic segmentation data of each part all contain their respective context semantic information.

[0114] In step 302, that is, calculate the attention weights of the target RGB image semantic segmentation data of different parts in the feature dimension and the spatial dimension to obtain the first feature attention weight and the first spatial attention weight. Calculate the attention weights of the target depth image semantic segmentation data of different parts in the feature dimension and the spatial dimension to obtain the second feature attention weight and the second spatial attention weight. Calculate the attention weights of the target three-dimensional point cloud semantic segmentation data in the feature dimension and the spatial dimension to obtain the third feature attention weight and the third spatial attention weight.

[0115] In steps 303 to 305, that is, use the self-attention mechanism to extract the corresponding features of the target RGB image data, the target depth image data, and the target three-dimensional point cloud data respectively.

[0116] In steps 306 to 310, that is, use the cross-attention mechanism to layer-fuse the features obtained by the self-attention mechanism to obtain the final three-dimensional reconstruction data of the tomato plant.

[0117] In the above steps, obtaining the three-dimensional reconstruction data of the tomato plant needs to satisfy the constraint that the first total loss function is minimized. The first total loss function can be expressed as:

[0118] ;

[0119] where is the photometric consistency loss, which is used to evaluate the photometric loss of multi-view data, is the semantic consistency loss, which is used to evaluate the semantic loss of multi-view data, is the multi-view adversarial loss, which is used to evaluate the adversarial loss of multi-view data, is the temporal consistency loss, which is used to evaluate the temporal loss of multi-view data, , , and are the weights of each loss.

[0120] In this embodiment, the target multi-modal data is first input into the semantic segmentation network so that the output target multi-modal semantic segmentation data has context semantic information. Then, the self-attention weights of the target multi-modal semantic segmentation data of different parts are calculated in the feature dimension and the spatial dimension. Based on the self-attention mechanism, the corresponding features of the target RGB image data, the target depth image data, and the target three-dimensional point cloud data are calculated respectively. Finally, the cross-attention mechanism is used to fuse the features layer by layer to obtain the three-dimensional reconstruction data of the tomato plant. Under the constraints of multiple losses, the multi-modal data from multiple perspectives are fully utilized for complementarity in terms of semantic segmentation, self-attention mechanism, and cross-attention mechanism, so as to be able to obtain the accurate three-dimensional reconstruction data of the tomato plant to the greatest extent.

[0121] Figure 4 It is the fourth flow schematic diagram of the greenhouse tomato plant growth parameter monitoring method provided by the embodiments of the present application. Referring to Figure 4 , in one embodiment, identifying the key growth parts of the tomato plant based on the three-dimensional reconstruction data may include:

[0122] 401. Input the three-dimensional reconstruction data into the Swin Transformer network to obtain the three-dimensional reconstruction features of multiple stages in the Swin Transformer network;

[0123] 402. Aggregate the three-dimensional reconstruction features of adjacent stages to obtain multiple first aggregation features;

[0124] 403. Group the multiple first aggregation features to obtain multiple groups of first aggregation features;

[0125] 404. Calculate the attention weights of each group of first aggregation features, and aggregate the multiple groups of first aggregation features based on the attention weights to obtain a second aggregation feature;

[0126] 405. Identify the key growth parts of the tomato plant based on the second aggregation feature.

[0127] In step 402, the aggregation of the three-dimensional reconstruction features of adjacent stages can be performed according to the following formula:

[0128] ;

[0129] Among them, is the first fusion feature of the th stage, is the three-dimensional reconstruction feature of the th stage, is the first fusion feature of the th stage, represents the upsampling operation, is a 1×1 convolutional layer.

[0130] This aggregation process enhances the information interaction between the high-level semantic features and the low-level detail features in the 3D reconstruction features, making the aggregated feature representation richer and more distinguishable.

[0131] In steps 403 to 404, after grouping multiple first aggregation features, the self-attention weights of each group are calculated based on the first aggregation features within each group, and then the first aggregation features of all groups are weighted and summed using the self-attention weights of each group, so as to obtain the second aggregation features with global semantic information.

[0132] This method significantly reduces the computational complexity of global attention by grouping, enabling it to be efficiently applied to large-scale feature maps and retaining more spatial details while capturing global semantic information.

[0133] In step 405, through two aggregations, sufficiently fine features of each part of the tomato plant are obtained, so that its various key growth parts can be accurately identified.

[0134] In the above steps, identifying the key growth parts of the tomato plant needs to satisfy the constraint of minimizing the second total loss function, and the second total loss function can be expressed as:

[0135] ;

[0136] where is the information noise contrastive estimation loss, which is used to evaluate the discriminability between foreground data and background data in multi-view data. The smaller this loss, the stronger the discriminability. is the relative position loss, which is used to evaluate the loss between the predicted distance and the true distance between the center point data of any two parts of the tomato plant in multi-view data. is the size ratio loss, which is used to evaluate the loss between the predicted size and the true size of any part of the tomato plant in multi-view data. 、 and are the weights of each loss.

[0137] In this embodiment, aiming at the characteristics of dense distribution of tomato plants and complex and changeable backgrounds in the greenhouse, under the constraints of foreground-background discriminability and prior knowledge of the distance and size between parts of tomato plants, first, the Swin Transformer (Shifted Window Transformer) network is used to extract three-dimensional reconstruction features at multiple stages from the three-dimensional reconstruction data. Then, based on upsampling and convolutional layers, the three-dimensional reconstruction features of adjacent stages are aggregated to extract richer and more distinguishable first aggregated features. Then, the multiple first aggregated features are grouped, and after obtaining the self-attention weights of each group, the first aggregated features of each group are re-aggregated to extract second aggregated features with global semantic information. At this time, the features of each part of the tomato plant are fine and obvious enough to accurately identify its key growth parts.

[0138] In one embodiment, based on the three-dimensional reconstruction data corresponding to the key growth parts, obtaining the target growth parameters of the key growth parts may include:

[0139] Inputting the three-dimensional reconstruction data corresponding to the key growth parts into the target growth parameter recognition model to obtain the target growth parameters output by the target growth parameter recognition model;

[0140] The key growth parts include the flowers of the tomato plant, the leaves of the tomato plant, the stems of the tomato plant, and the vertices of the tomato plant. The target growth parameters include the growth stage category of the flowers, the length of the leaves, the stem diameter of the stems, and the height of the vertices;

[0141] The target growth parameter recognition model is trained by using the historical three-dimensional reconstruction data of the key growth parts and their corresponding target growth parameter labels to train the basic model that matches the recognition of the target growth parameters.

[0142] Specifically, the target growth parameter recognition model may include a growth stage category recognition model, a length recognition model, a stem diameter recognition model, and a height recognition model, where:

[0143] 1. The growth stage category recognition model can be expressed as follows:

[0144] ;

[0145] Among them, is the growth stage type of the flower, including the bud stage, the flowering stage, and the withering stage, is the activation function, is the growth stage weight matrix of the fully connected layer. Introducing this weight matrix can assign different weights to flowers in different growth stages, so as to more accurately depict the dynamic process of the flower life cycle, is the max pooling operation, is the three-dimensional convolutional layer, The three-dimensional reconstruction data corresponding to the flower.

[0146] The historical three-dimensional reconstruction data of the flower for training the above model needs to cover different growth stages of the flower, so that the model can identify different growth stage categories of the flower.

[0147] 2. The length recognition model can be expressed as follows:

[0148] ;

[0149] Among them, is the length of the leaf, is the attention pooling layer, is the multi-scale grouping operation, is the backbone network for point cloud data processing, is the three-dimensional reconstruction data corresponding to the leaf.

[0150] This model can achieve the recognition of leaf length end-to-end without explicitly extracting the leaf skeleton and fitting the vein curve, greatly improving the robustness and efficiency of recognition.

[0151] 3. The stem diameter recognition model can be expressed as follows:

[0152] ;

[0153] ;

[0154] ;

[0155] Among them, is the set of three-dimensional reconstruction data in the local neighborhood centered on the th cross-section of the main stem and in the direction of the th perspective, is the data coordinate in the set, is the radius for circle fitting of is the center coordinate for circle fitting of is the data coordinate and radius obtained after circle fitting, is the stem diameter at the th cross-section of the main stem and the th perspective, is the weight of

[0156] is the weighted average of the stem diameters at all cross-sections and all perspectives of the main stem, that is, the stem diameter finally output by the model.

[0156] It should be noted that all cross-sections of the main stem are measured at equal intervals in the direction of the main stem. It can be adaptively determined according to the viewing angle and the measurement quality of the cross-section, and the measurement quality can be evaluated by factors such as fitting residuals and data density.

[0157] This model integrates the stem diameter estimation of multiple cross-sections and multiple viewing angles, effectively reducing the influence of local occlusion and noise, and improving the accuracy and stability of stem diameter estimation.

[0158] 4. The height recognition model can be expressed as follows:

[0159] ;

[0160] Where is the height of the vertex at time, is the 3D reconstruction data of the vertex 's maximum height, is the relative growth rate of the maximum height, is the time when the relative growth rate of the maximum height is the largest.

[0161] In this embodiment, for different key growth parts, a differentiated target growth parameter recognition model is used to identify their target growth parameters: for the flowers of tomato plants, a lightweight convolutional neural network is used, with the 3D reconstruction data of the flowers as the input, and through 3D feature extraction and fully connected layer mapping, the growth period category of the flowers is identified; for the leaves of tomato plants, PointNet++ backbone network, with the 3D reconstruction data of the leaves as the input, and through multi-scale feature learning and attention mechanism, the length of the leaves is identified; for the stems of tomato plants, a circle fitting method is used, with the 3D reconstruction data of multiple cross-sections and multiple viewing angles of the main stem as the input, and through the integration of the stem diameter estimation of multiple cross-sections and multiple viewing angles, the stem diameter of the stem is identified; for the vertices of tomato plants, a growth curve fitting method is used, with the 3D reconstruction data of the vertices as the input, and through fitting the vertex growth curve, the height of the vertices is identified. In this embodiment, according to the characteristics of different key growth parts, different methods are used to fully extract their 3D features from the 3D reconstruction data, so as to accurately identify their target growth parameters.

[0162] Figure 5 is the fifth schematic flow chart of the greenhouse tomato plant growth parameter monitoring method provided by the embodiments of the present application. Referring to Figure 5 , in one embodiment, based on the current time series data of the target growth parameter, obtaining the future predicted value of the target growth parameter may include:

[0163] 501. Based on the current time series data of the target growth parameter, generate the lag time series data and the moving average time series data of the target growth parameter;

[0164] 502. Concatenate the current time series data, lagged time series data, and moving average time series data to obtain concatenated time series data;

[0165] 503. Perform convolution and pooling on the concatenated time series data to obtain multiple local features of the target growth parameter in the spatio-temporal dimension;

[0166] 504. Capture the long-term dependence relationships between each local feature and the local features in the temporal sequences before and after it to obtain multiple long-term dependence local features of the target growth parameter in the spatio-temporal dimension;

[0167] 505. Calculate the attention weights between the multiple long-term dependence local features;

[0168] 506. Based on the attention weights, calculate the hidden state representations of the target growth parameter in the spatio-temporal dimension with each long-term dependence local feature as the center to obtain multiple hidden state representations;

[0169] 507. Map the multiple hidden state representations to the output space to obtain the future predicted values of the target growth parameter.

[0170] For the aforementioned length recognition model, stem diameter recognition model, and height recognition model, generally, they can only identify the current target growth parameter based on the current three-dimensional reconstruction data corresponding to the key growth parts of the input, and cannot predict the future target growth parameter. In this embodiment, the time series data obtained through multiple recognitions can be used to achieve the prediction of the future target growth parameter.

[0171] In steps 502 to 503, for any target growth parameter, its current time series data , lagged time series data , and moving average time series data can be concatenated, convolved, and pooled according to the following formula to obtain a set of multiple local features of the target growth parameter in the spatio-temporal dimension :

[0172] ;

[0173] where is a one-dimensional max pooling operation, is an activation function, is a one-dimensional convolutional layer, and includes local features of multiple time steps.

[0174] In step 504, for the local feature in the time step in , the following formula can be used to capture the long-term dependence relationship between it and the local features in the temporal sequences before and after it:

[0175] ;

[0176] ;

[0177] Among them, is the local feature at time step and the long-term dependence representation between its previous local features, is the long-term dependence representation between the local feature at time step and the local feature at time step and ; is the long-term dependence representation between the local feature at time step and its subsequent local features, is the long-term dependence representation between the local feature at time step and the local feature at time step and ; is a Bidirectional Gated Recurrent Unit (BiGRU) network.

[0178] Then, the long-term dependence local features of the target growth parameter in the spatio-temporal dimension are obtained according to the following formula :

[0179] ;

[0180] In step 506, taking as an example, using the attention weights between it and other long-term dependence local features, other long-term dependence local features are weighted and summed to obtain a hidden state representation of the target growth parameter in the spatio-temporal dimension. Similarly, taking other long-term dependence local features as examples, other hidden state representations of the target growth parameter in the spatio-temporal dimension can be obtained.

[0181] In step 507, these hidden state representations are mapped to the output space through a fully connected layer to obtain the future predicted value of the target growth parameter.

[0182] In the above steps, obtaining the future predicted value of the target growth parameter needs to satisfy the constraint of minimizing the asymmetric loss function, and the asymmetric loss function can be expressed as :

[0183] ;

[0184] Among them, is the future predicted value, is the future actual value, and are the penalty coefficients for overestimation and underestimation respectively, which can be set according to actual needs and are not limited here.

[0185] In this embodiment, various time-series data of the target growth parameters are first spliced to obtain spliced time-series data, and then one-dimensional convolution and one-dimensional pooling operations in the convolutional neural network are used to automatically learn the high-order patterns in the spliced time-series data to extract the local features of the target growth parameters in the spatio-temporal dimension. Then, the bidirectional gated recurrent unit network is used to learn the long-term dependence between adjacent time-series local features, and then multiple hidden state representations of the target growth parameters in the spatio-temporal dimension are calculated according to the attention weights, so as to be mapped to the output space to obtain the future predicted value of the target growth parameters. Through the hybrid architecture of the convolutional neural network and the bidirectional gated recurrent unit network, hierarchical feature learning enhances the representation ability of complex spliced time-series data, so that accurate future predicted values of the target growth parameters can be obtained.

[0186] In one embodiment, preprocessing the initial multi-modal data to obtain target multi-modal data may include:

[0187] 1. Perform exposure correction, image enhancement, distortion correction, and denoising on the initial RGB image data in sequence to obtain the RGB image data to be processed. Specifically:

[0188] 1.1. Use the automatic exposure algorithm of the RGB camera to correct the exposure time of the initial RGB image data to obtain the first corrected RGB image data;

[0189] 1.2. Use the adaptive histogram equalization algorithm to perform pixel-level image enhancement on the first corrected RGB image data to obtain the second corrected RGB image data;

[0190] The adaptive histogram equalization algorithm performs histogram equalization on the first corrected RGB image data in each local area by performing block processing on the first corrected RGB image data, which not only improves the contrast of the image data but also preserves local details;

[0191] 1.3. Use the wide-angle Brown-Conrady distortion model to perform pixel-level lens distortion correction on the second corrected RGB image data to obtain the third corrected RGB image data;

[0192] 1.4. Use guided filtering to denoise the third corrected RGB image data, smoothing the image data while maintaining the image edge data to obtain the RGB image data to be processed.

[0193] 2. Perform invalid value filling and denoising on the initial depth image data in sequence to obtain the depth image data to be processed;

[0194] 3. Remove outliers, downsample, and estimate normal vectors from the initial 3D point cloud data in sequence to obtain the 3D point cloud data to be processed;

[0195] After the above preprocessing is completed, align the RGB image data to be processed, the depth image data to be processed, and the 3D point cloud data to be processed to obtain the target RGB image data, the target depth image data, and the target 3D point cloud data. Specifically:

[0196] Estimate the internal parameters of the RGB camera and the depth camera using the Zhang Zhengyou calibration method. Transform the RGB image data to be processed into the RGB camera coordinate system using the internal parameters of the RGB camera, and transform the depth image data to be processed into the depth camera coordinate system using the internal parameters of the depth camera;

[0197] Use the Iterative Closest Point (ICP) algorithm to register the black and white checkerboard planes in the RGB camera coordinate system with the black and white checkerboard planes in the LiDAR sensor coordinate system, and the black and white checkerboard planes in the depth camera coordinate system with the black and white checkerboard planes in the LiDAR sensor coordinate system, respectively, to establish the transformation external parameters between the two camera coordinate systems and the LiDAR sensor coordinate system;

[0198] Based on the following formula, transform each frame of the 3D point cloud data to be processed from the LiDAR sensor coordinate system to the RGB camera coordinate system and the depth camera coordinate system respectively:

[0199] ;

[0200] where is the pixel coordinate of the 3D point cloud data to be processed in the RGB camera coordinate system or the depth camera coordinate system, is the corresponding transformation external parameter, is the homogeneous coordinate of the 3D point cloud data to be processed in the LiDAR sensor coordinate system.

[0201] In this embodiment, by performing targeted preprocessing on the initial RGB image data, the initial depth image data, and the initial 3D point cloud data, and aligning the data, the data quality can be improved, which is helpful for the effective fusion of subsequent target multi-modal data.

[0202] Next, the greenhouse tomato plant growth parameter monitoring device provided by the embodiments of the present application will be described. The greenhouse tomato plant growth parameter monitoring device described below can be mutually referred to with the greenhouse tomato plant growth parameter monitoring method described above.

[0203] Figure 6 is the structural schematic diagram of the greenhouse tomato plant growth parameter monitoring device provided by the embodiments of the present application. Refer to Figure 6, an embodiment of the present application provides a monitoring device for growth parameters of greenhouse tomato plants, which may include:

[0204] An initial multimodal data acquisition module 601, configured to: acquire initial multimodal data of a tomato plant from multiple perspectives collected by a collection trolley in a target greenhouse; the initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data;

[0205] A target multimodal data acquisition module 602, configured to: preprocess the initial multimodal data to obtain target multimodal data;

[0206] A three-dimensional reconstruction data acquisition module 603, configured to perform fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant;

[0207] A key growth part identification module 604, configured to identify key growth parts of the tomato plant based on the three-dimensional reconstruction data;

[0208] A target growth parameter acquisition module 605, configured to obtain target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts;

[0209] A target growth parameter prediction module 606, configured to obtain future predicted values of the target growth parameters based on current time series data of the target growth parameters.

[0210] The monitoring device for growth parameters of greenhouse tomato plants provided in this embodiment acquires initial multimodal data of a tomato plant from multiple perspectives collected by a collection trolley in a target greenhouse. The initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data. The initial multimodal data is preprocessed to obtain target multimodal data, and the target multimodal data is subjected to fusion modeling to obtain three-dimensional reconstruction data of the tomato plant. Key growth parts of the tomato plant are identified based on the three-dimensional reconstruction data, and target growth parameters of the key growth parts are obtained based on the three-dimensional reconstruction data corresponding to the key growth parts. Future predicted values of the target growth parameters are obtained based on current time series data of the target growth parameters. By acquiring RGB image data, depth image data, and three-dimensional point cloud data of the tomato plant from multiple perspectives in this embodiment, on the one hand, the data acquisition dimension is increased, and on the other hand, through multi-perspective acquisition, the influence of mutual overlap and occlusion between plants on the integrity of data acquisition can be avoided. After three-dimensional reconstruction of the plant based on these data, the key parts of the plant can be accurately identified and their target growth parameters can be obtained, and by predicting the future values through the current time series data of the target growth parameters, dynamic growth data of the target growth parameters can be obtained, thereby realizing dynamic monitoring of the target growth parameters of the plant.

[0211] In one embodiment, it further includes an initial multi-modal data acquisition module (not shown in the figure) for:

[0212] Based on the positions of multiple tomato plants in the target greenhouse, the already acquired positions adjacent to the multiple tomato plants, the position of the acquisition trolley, and the speed of the acquisition trolley, construct the target path function of the acquisition trolley; the speed includes linear speed and angular speed;

[0213] Adjust the linear speed and angular speed of the acquisition trolley until the value of the target path function is minimized, and determine the linear speed and angular speed of the acquisition trolley at this time as the optimal linear speed and optimal angular speed;

[0214] Based on the optimal angular speed, obtain the optimal heading angle of the acquisition trolley;

[0215] Based on the optimal linear speed and the optimal heading angle, control the acquisition trolley to move and acquire the multi-modal data of each tomato plant in the target greenhouse, and obtain the initial multi-modal data from multiple perspectives.

[0216] In one embodiment, the three-dimensional reconstruction data acquisition module 603 is specifically used for:

[0217] Input the target multi-modal data into a pre-trained semantic segmentation network to obtain the target multi-modal semantic segmentation data of different parts of the tomato plants output by the pre-trained semantic segmentation network; the target multi-modal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data, and target three-dimensional point cloud semantic segmentation data;

[0218] Calculate the attention weights of the target multi-modal semantic segmentation data of different parts in the feature dimension and the spatial dimension to obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data;

[0219] Based on the first feature attention weight and the first spatial attention weight, obtain the texture feature of the target RGB image data;

[0220] Based on the second feature attention weight and the second spatial attention weight, obtain the depth feature of the target depth image data;

[0221] Based on the third feature attention weight and the third spatial attention weight, obtain the normal vector feature of the target three-dimensional point cloud data;

[0222] Calculate the attention weights between every two of the texture feature, the depth feature, and the normal vector feature to obtain a first correlation weight between the texture feature and the depth feature, a second correlation weight between the texture feature and the normal vector feature, and a third correlation weight between the depth feature and the normal vector feature;

[0223] Based on the first correlation weight and the second correlation weight, obtain a first fusion feature with the texture feature as the central feature;

[0224] Based on the first correlation weight and the third correlation weight, obtain a second fusion feature with the depth feature as the central feature;

[0225] Based on the second correlation weight and the third correlation weight, obtain a third fusion feature with the normal vector feature as the central feature;

[0226] Perform weighted summation on the first fusion feature, the second fusion feature, and the third fusion feature to obtain the three-dimensional reconstruction data of the tomato plant.

[0227] In one embodiment, the key growth part recognition module 604 is specifically configured to:

[0228] Input the three-dimensional reconstruction data into the Swin Transformer network to obtain three-dimensional reconstruction features at multiple stages in the Swin Transformer network;

[0229] Aggregate the three-dimensional reconstruction features at adjacent stages to obtain a plurality of first aggregation features;

[0230] Group the plurality of first aggregation features to obtain multiple groups of first aggregation features;

[0231] Calculate the attention weights of each group of first aggregation features, and aggregate the multiple groups of first aggregation features based on the attention weights to obtain a second aggregation feature;

[0232] Identify the key growth parts of the tomato plant based on the second aggregation feature.

[0233] In one embodiment, the target growth parameter acquisition module 605 is specifically configured to:

[0234] Input the three-dimensional reconstruction data corresponding to the key growth parts into the target growth parameter recognition model to obtain the target growth parameters output by the target growth parameter recognition model;

[0235] The key growth parts include the flowers of the tomato plant, the leaves of the tomato plant, the stems of the tomato plant, and the apex of the tomato plant, and the target growth parameters include the growth period category of the flowers, the length of the leaves, the stem diameter of the stems, and the height of the apex;

[0236] The target growth parameter recognition model is obtained by training a basic model that matches the recognition of the target growth parameters with the historical three-dimensional reconstruction data of the key growth parts and their corresponding target growth parameter labels.

[0237] In one embodiment, the target growth parameter prediction module 606 is specifically configured to:

[0238] Generate lagged time series data and moving average time series data of the target growth parameters based on the current time series data of the target growth parameters;

[0239] Concatenate the current time series data, the lagged time series data, and the moving average time series data to obtain concatenated time series data;

[0240] Perform convolution and pooling on the concatenated time series data to obtain multiple local features of the target growth parameters in the spatio-temporal dimension;

[0241] Capture the long-term dependence relationship between each local feature and the local features in the previous and subsequent time series to obtain multiple long-term dependence local features of the target growth parameters in the spatio-temporal dimension;

[0242] Calculate the attention weights between the multiple long-term dependence local features;

[0243] Based on the attention weights, calculate the hidden state representation of the target growth parameters in the spatio-temporal dimension with each long-term dependence local feature as the center to obtain multiple hidden state representations;

[0244] Map the multiple hidden state representations to the output space to obtain the future prediction values of the target growth parameters.

[0245] In one embodiment, the target multi-modal data acquisition module 602 is specifically configured to:

[0246] Perform exposure correction, image enhancement, distortion correction, and denoising on the initial RGB image data in sequence to obtain the RGB image data to be processed;

[0247] Perform invalid value filling and denoising on the initial depth image data in sequence to obtain the depth image data to be processed;

[0248] Perform outlier removal, downsampling, and normal vector estimation on the initial three-dimensional point cloud data in sequence to obtain the three-dimensional point cloud data to be processed;

[0249] Align the to-be-processed RGB image data, the to-be-processed depth image data, and the to-be-processed three-dimensional point cloud data to obtain target RGB image data, target depth image data, and target three-dimensional point cloud data.

[0250] FIG. 7 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 7 shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call a computer program in the memory 730 to execute the steps of the method for monitoring the growth parameters of greenhouse tomato plants, for example, including:

[0251] Obtain the initial multi-modal data of the tomato plants collected by the acquisition trolley in the target greenhouse from multiple perspectives; the initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data;

[0252] Preprocess the initial multi-modal data to obtain target multi-modal data;

[0253] Perform fusion modeling on the target multi-modal data to obtain the three-dimensional reconstruction data of the tomato plants;

[0254] Identify the key growth parts of the tomato plants based on the three-dimensional reconstruction data;

[0255] Obtain the target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts;

[0256] Obtain the future predicted values of the target growth parameters based on the current time series data of the target growth parameters.

[0257] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0258] On the other hand, an embodiment of this application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the greenhouse tomato plant growth parameter monitoring method provided in the above-mentioned various embodiments, for example, including:

[0259] Obtain the initial multi-modal data of the tomato plants at multiple perspectives collected by the acquisition trolley in the target greenhouse; the initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data;

[0260] Preprocess the initial multi-modal data to obtain target multi-modal data;

[0261] Perform fusion modeling on the target multi-modal data to obtain the three-dimensional reconstruction data of the tomato plants;

[0262] Identify the key growth parts of the tomato plants based on the three-dimensional reconstruction data;

[0263] Based on the three-dimensional reconstruction data corresponding to the key growth parts, obtain the target growth parameters of the key growth parts;

[0264] Based on the current time series data of the target growth parameters, obtain the future predicted values of the target growth parameters.

[0265] On the other hand, an embodiment of this application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. The computer program is used to cause a processor to execute the steps of the greenhouse tomato plant growth parameter monitoring method provided in the above-mentioned various embodiments, for example, including:

[0266] Obtain the initial multi-modal data of tomato plants collected by the acquisition trolley in the target greenhouse from multiple perspectives; the initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data;

[0267] Preprocess the initial multi-modal data to obtain target multi-modal data;

[0268] Perform fusion modeling on the target multi-modal data to obtain the three-dimensional reconstruction data of the tomato plants;

[0269] Identify the key growth parts of the tomato plants based on the three-dimensional reconstruction data;

[0270] Based on the three-dimensional reconstruction data corresponding to the key growth parts, obtain the target growth parameters of the key growth parts;

[0271] Based on the current time-series data of the target growth parameters, obtain the future predicted values of the target growth parameters.

[0272] The non-transitory computer-readable storage medium can be any available medium or data storage device accessible by the processor, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical discs (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROM, EPROM, EEPROM, non-volatile memories (NAND FLASH), solid-state drives (SSD)), etc.

[0273] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0274] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0275] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A method for monitoring the growth parameters of greenhouse tomato plants, characterized in that, Including: Obtaining initial multi-modal data of tomato plants collected by a collection trolley in a target greenhouse from multiple perspectives; The initial multi-modal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; the initial RGB image data, the initial depth image data, and the initial three-dimensional point cloud data are multi-perspective data of tomato plants collected by an RGB camera, a depth camera, and a LiDAR sensor on the collection trolley respectively; in the driving route planning of the collection trolley, the distance between the position of the collection trolley and the position of the collected data adjacent to each tomato plant is used as a penalty term; Preprocessing the initial multi-modal data to obtain target multi-modal data; Performing fusion modeling on the target multi-modal data to obtain three-dimensional reconstruction data of the tomato plants; Identifying key growth parts of the tomato plants based on the three-dimensional reconstruction data; Obtaining target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts; Obtaining future predicted values of the target growth parameters based on current time series data of the target growth parameters, including: Generating lag time series data and moving average time series data of the target growth parameters based on the current time series data of the target growth parameters; Concatenating the current time series data, the lag time series data, and the moving average time series data to obtain concatenated time series data; Performing convolution and pooling on the concatenated time series data to obtain multiple local features of the target growth parameters in the spatio-temporal dimension; Capturing the long-term dependence relationship between each local feature and local features in the front and back time series to obtain multiple long-term dependence local features of the target growth parameters in the spatio-temporal dimension; Calculating attention weights between multiple long-term dependence local features; Based on the attention weights, calculating the hidden state representation of the target growth parameters in the spatio-temporal dimension with each long-term dependence local feature as the center to obtain multiple hidden state representations; Mapping the multiple hidden state representations to the output space to obtain future predicted values of the target growth parameters.

2. The method for monitoring the growth parameters of greenhouse tomato plants according to claim 1, characterized in that, The initial multi-modal data from multiple perspectives is collected based on the following method: Based on the positions of multiple tomato plants in the target greenhouse, the already collected positions adjacent to the multiple tomato plants, the position of the collection trolley, and the speed of the collection trolley, constructing a target path function of the collection trolley; the speed includes linear speed and angular speed; Adjusting the linear speed and angular speed of the collection trolley until the value of the target path function is minimized, and determining the linear speed and angular speed of the collection trolley at this time as the optimal linear speed and optimal angular speed; Obtaining the optimal heading angle of the collection trolley based on the optimal angular speed; Based on the optimal linear speed and the optimal heading angle, controlling the movement of the collection trolley and collecting multi-modal data of each tomato plant in the target greenhouse to obtain the initial multi-modal data from multiple perspectives.

3. The method for monitoring the growth parameters of greenhouse tomato plants according to claim 1, wherein, The performing fusion modeling on the target multi-modal data to obtain three-dimensional reconstruction data of the tomato plants includes: Input the target multi-modal data into a pre-trained semantic segmentation network to obtain the target multi-modal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network; the target multi-modal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data, and target 3D point cloud semantic segmentation data; Calculate the attention weights of the target multi-modal semantic segmentation data of different parts in the feature dimension and the spatial dimension to obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target 3D point cloud semantic segmentation data; Based on the first feature attention weight and the first spatial attention weight, obtain the texture feature of the target RGB image data; Based on the second feature attention weight and the second spatial attention weight, obtain the depth feature of the target depth image data; Based on the third feature attention weight and the third spatial attention weight, obtain the normal vector feature of the target 3D point cloud data; Calculate the attention weights between every two of the texture feature, the depth feature, and the normal vector feature to obtain the first correlation weight between the texture feature and the depth feature, the second correlation weight between the texture feature and the normal vector feature, and the third correlation weight between the depth feature and the normal vector feature; Based on the first correlation weight and the second correlation weight, obtain the first fusion feature with the texture feature as the central feature; Based on the first correlation weight and the third correlation weight, obtain the second fusion feature with the depth feature as the central feature; Based on the second correlation weight and the third correlation weight, obtain the third fusion feature with the normal vector feature as the central feature; Perform weighted summation on the first fusion feature, the second fusion feature, and the third fusion feature to obtain the 3D reconstruction data of the tomato plant.

4. The method for monitoring the growth parameters of greenhouse tomato plants according to claim 1, characterized in that, The identifying the key growth parts of the tomato plant based on the 3D reconstruction data includes: Input the 3D reconstruction data into the Swin Transformer network to obtain the 3D reconstruction features of multiple stages in the Swin Transformer network; Aggregate the 3D reconstruction features of adjacent stages to obtain multiple first aggregation features; Group the multiple first aggregation features to obtain multiple groups of first aggregation features; Calculate the attention weights of each group of first aggregation features, and aggregate the multiple groups of first aggregation features based on the attention weights to obtain the second aggregation feature; Identify the key growth parts of the tomato plant based on the second aggregation feature.

5. The method for monitoring the growth parameters of greenhouse tomato plants according to claim 1, characterized in that, The obtaining the target growth parameters of the key growth parts based on the 3D reconstruction data corresponding to the key growth parts includes: Input the three-dimensional reconstruction data corresponding to the key growth parts into the target growth parameter recognition model to obtain the target growth parameters output by the target growth parameter recognition model; The key growth parts include the flowers of the tomato plant, the leaves of the tomato plant, the stems of the tomato plant, and the apex of the tomato plant, and the target growth parameters include the growth period category of the flowers, the length of the leaves, the stem diameter of the stems, and the height of the apex; The target growth parameter recognition model is obtained by training a basic model that matches the target growth parameter recognition with the historical three-dimensional reconstruction data of the key growth parts and their corresponding target growth parameter labels.

6. The method for monitoring the growth parameters of greenhouse tomato plants according to claim 1, wherein The preprocessing of the initial multimodal data to obtain the target multimodal data includes: Perform exposure correction, image enhancement, distortion correction, and denoising on the initial RGB image data in sequence to obtain the RGB image data to be processed; Perform invalid value filling and denoising on the initial depth image data in sequence to obtain the depth image data to be processed; Perform outlier removal, downsampling, and normal vector estimation on the initial three-dimensional point cloud data in sequence to obtain the three-dimensional point cloud data to be processed; Align the RGB image data to be processed, the depth image data to be processed, and the three-dimensional point cloud data to be processed to obtain the target RGB image data, the target depth image data, and the target three-dimensional point cloud data.

7. A monitoring device for growth parameters of greenhouse tomato plants, characterized in that, Including: An initial multimodal data acquisition module, configured to: acquire the initial multimodal data of the tomato plant from multiple perspectives collected by the acquisition trolley in the target greenhouse; The initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; the initial RGB image data, the initial depth image data, and the initial three-dimensional point cloud data are the multi-perspective data of the tomato plant collected by the RGB camera, the depth camera, and the LiDAR sensor on the acquisition trolley respectively; in the driving route planning of the acquisition trolley, the distance between the position of the acquisition trolley and the position of the collected data adjacent to each tomato plant is used as a penalty term; A target multimodal data acquisition module, configured to: preprocess the initial multimodal data to obtain the target multimodal data; A three-dimensional reconstruction data acquisition module, configured to perform fusion modeling on the target multimodal data to obtain the three-dimensional reconstruction data of the tomato plant; A key growth part recognition module, configured to recognize the key growth parts of the tomato plant based on the three-dimensional reconstruction data; A target growth parameter acquisition module, configured to obtain the target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts; A target growth parameter prediction module, configured to obtain the future prediction values of the target growth parameters based on the current time series data of the target growth parameters, including: Generate the lag time series data and the moving average time series data of the target growth parameters based on the current time series data of the target growth parameters; Concatenate the current time series data, the lag time series data, and the moving average time series data to obtain the concatenated time series data; Perform convolution and pooling on the spliced time-series data to obtain multiple local features of the target growth parameter in the spatio-temporal dimension; Capture the long-term dependence relationships between each local feature and the local features in the time series before and after it to obtain multiple long-term dependence local features of the target growth parameter in the spatio-temporal dimension; Calculate the attention weights between multiple long-term dependence local features; Based on the attention weights, calculate the hidden state representation of the target growth parameter in the spatio-temporal dimension with each long-term dependence local feature as the center to obtain multiple hidden state representations; Map the multiple hidden state representations to the output space to obtain the future predicted values of the target growth parameter.

8. An electronic device, comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the greenhouse tomato plant growth parameter monitoring method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the greenhouse tomato plant growth parameter monitoring method according to any one of claims 1 to 6.

Images