Greenhouse tomato plant growth parameter monitoring method, device, equipment and product

Abstract

The invention relates to the technical field of agriculture, and provides a greenhouse tomato plant growth parameter monitoring method, device, equipment and product. The method comprises the following steps: preprocessing initial multi-modal data of a tomato plant under multiple view angles; carrying out fusion modeling on the obtained target multi-modal data to obtain three-dimensional reconstruction data of the tomato plant; identifying a key growth part of the tomato plant based on the three-dimensional reconstruction data, and obtaining a target growth parameter of the key growth part; and obtaining a future predicted value of the target growth parameter based on the current time sequence data of the target growth parameter. According to the method, the data acquisition dimension is increased, the influence of mutual overlapping and shielding between plants on the data is avoided through multi-view acquisition, and after three-dimensional reconstruction is performed based on the data, the key parts of the plants can be accurately identified and the target growth parameters of the plants can be acquired; and the future value of the target growth parameter is predicted through the current time sequence data of the target growth parameter, so that the target growth parameter is dynamically monitored.

Description

Technical Field

[0001] The present application relates to the field of agricultural technology, and in particular to a method, device, equipment and product for monitoring growth parameters of greenhouse tomato plants. Background Art

[0002] With the rapid development of modern agriculture, greenhouse horticulture has become an important way to improve the comprehensive agricultural production capacity. As a representative crop of greenhouse horticulture, the yield and quality of greenhouse tomatoes are directly related to the process of agricultural industrialization. Accurately monitoring the growth status of tomato plants is the key to realizing digital and intelligent management of tomato production.

[0003] However, the current greenhouse tomato plant growth monitoring mainly relies on manual monitoring and semi-automatic monitoring. Manual monitoring relies on experience to visually observe and simply measure the growth status of plants, which has the disadvantages of low monitoring efficiency and accuracy. Although semi-automatic monitoring introduces machine vision analysis to extract some growth parameters of plants, most of them are based on two-dimensional images with a single feature dimension. In addition, since plants often overlap and occlude to varying degrees during their growth, it is difficult for two-dimensional images to accurately obtain the complete growth parameters of plants, thus affecting the monitoring accuracy. Furthermore, whether it is manual monitoring or semi-automatic monitoring, they can only monitor the current growth parameters of plants based on current visual observation or two-dimensional images, but cannot dynamically monitor the continuous changes in growth parameters of plants during their growth.

[0004] In summary, the current greenhouse tomato plant growth monitoring method is based on manual monitoring and semi-automatic monitoring, which results in low monitoring efficiency and accuracy, and is unable to dynamically monitor the continuous changes in growth parameters of plants during the growth process. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and product for monitoring the growth parameters of greenhouse tomato plants, which are used to solve the technical problems that the current greenhouse tomato plant growth monitoring method is based on manual monitoring and semi-automatic monitoring, resulting in low monitoring efficiency and monitoring accuracy, and cannot dynamically monitor the continuous changes in growth parameters of plants during the growth process.

[0006] In a first aspect, an embodiment of the present application provides a method for monitoring growth parameters of greenhouse tomato plants, comprising: Acquire initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple viewing angles; the initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; Preprocessing the initial multimodal data to obtain target multimodal data; Performing fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; identifying key growth parts of the tomato plant based on the three-dimensional reconstruction data; Based on the three-dimensional reconstruction data corresponding to the key growth part, a target growth parameter of the key growth part is obtained; Based on the current time series data of the target growth parameter, a future predicted value of the target growth parameter is obtained.

[0007] In one embodiment, the initial multimodal data under multiple viewing angles is collected based on the following method: Based on the positions of the plurality of tomato plants in the target greenhouse, the adjacent collected positions of the plurality of tomato plants, the position of the collection vehicle and the speed of the collection vehicle, a target path function of the collection vehicle is constructed; the speed includes a linear speed and an angular speed; Adjusting the linear speed and angular speed of the collection vehicle until the value of the target path function is minimized, and determining the linear speed and angular speed of the collection vehicle at this time as the optimal linear speed and optimal angular speed; Based on the optimal angular velocity, an optimal heading angle of the collection vehicle is obtained; Based on the optimal linear speed and the optimal heading angle, the collection vehicle is controlled to move and collect multimodal data of each tomato plant in the target greenhouse to obtain initial multimodal data under the multiple viewing angles.

[0008] In one embodiment, the fusion modeling of the target multimodal data to obtain the three-dimensional reconstruction data of the tomato plant includes: Inputting the target multimodal data into a pre-trained semantic segmentation network to obtain target multimodal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network; the target multimodal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data and target three-dimensional point cloud semantic segmentation data; Calculate the attention weights of the target multimodal semantic segmentation data of the different parts in the feature dimension and the spatial dimension, and obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data; Obtaining texture features of target RGB image data based on the first feature attention weight and the first spatial attention weight; Obtaining a depth feature of the target depth image data based on the second feature attention weight and the second spatial attention weight; Based on the third feature attention weight and the third spatial attention weight, obtaining a normal vector feature of the target three-dimensional point cloud data; Calculating the attention weight between each two features among the texture feature, the depth feature and the normal vector feature to obtain a first association weight between the texture feature and the depth feature, a second association weight between the texture feature and the normal vector feature, and a third association weight between the depth feature and the normal vector feature; Based on the first association weight and the second association weight, obtaining a first fusion feature with the texture feature as a central feature; Based on the first association weight and the third association weight, obtaining a second fusion feature with the depth feature as a central feature; Based on the second association weight and the third association weight, obtaining a third fusion feature with the normal vector feature as a central feature; The first fusion feature, the second fusion feature and the third fusion feature are weightedly summed to obtain the three-dimensional reconstruction data of the tomato plant.

[0009] In one embodiment, the identifying the key growth parts of the tomato plant based on the three-dimensional reconstruction data includes: Inputting the three-dimensional reconstruction data into a Swin Transformer network to obtain three-dimensional reconstruction features of multiple stages in the Swin Transformer network; Aggregating the three-dimensional reconstruction features of adjacent stages to obtain a plurality of first aggregated features; Grouping the multiple first aggregate features to obtain multiple groups of first aggregate features; Calculating the attention weight of each group of first aggregate features, and aggregating the multiple groups of first aggregate features based on the attention weight to obtain a second aggregate feature; Key growth parts of the tomato plant are identified based on the second aggregated feature.

[0010] In one embodiment, obtaining the target growth parameters of the key growth part based on the three-dimensional reconstruction data corresponding to the key growth part includes: Inputting the three-dimensional reconstruction data corresponding to the key growth part into a target growth parameter recognition model to obtain the target growth parameter output by the target growth parameter recognition model; The key growth parts include the flower of the tomato plant, the leaves of the tomato plant, the stem of the tomato plant and the apex of the tomato plant, and the target growth parameters include the growth period category of the flower, the length of the leaves, the stem thickness of the stem and the height of the apex; The target growth parameter identification model is obtained by training a basic model that matches the target growth parameter identification through the historical three-dimensional reconstruction data of the key growth part and its corresponding target growth parameter label.

[0011] In one embodiment, obtaining a future predicted value of the target growth parameter based on the current time series data of the target growth parameter includes: Based on the current time series data of the target growth parameter, generating the lagged time series data and the moving average time series data of the target growth parameter; splicing the current time series data, the lagged time series data and the moving average time series data to obtain spliced ​​time series data; Convolution and pooling are performed on the spliced ​​time series data to obtain multiple local features of the target growth parameter in the spatiotemporal dimension; Capturing the long-term dependency relationship between each local feature and the local features before and after it, and obtaining a plurality of long-term dependent local features of the target growth parameter in the spatiotemporal dimension; Calculate the attention weights between multiple long-term dependent local features; Based on the attention weights, a hidden state representation of the target growth parameter in the spatiotemporal dimension is calculated with each long-term dependent local feature as the center to obtain multiple hidden state representations; The multiple hidden state representations are mapped to an output space to obtain future predicted values ​​of the target growth parameters.

[0012] In one embodiment, preprocessing the initial multimodal data to obtain target multimodal data includes: Performing exposure correction, image enhancement, distortion correction and denoising on the initial RGB image data in sequence to obtain RGB image data to be processed; Performing invalid value filling and denoising on the initial depth image data in sequence to obtain depth image data to be processed; Performing outlier removal, downsampling, and normal vector estimation on the initial three-dimensional point cloud data in sequence to obtain three-dimensional point cloud data to be processed; The RGB image data to be processed, the depth image data to be processed and the three-dimensional point cloud data to be processed are aligned to obtain target RGB image data, target depth image data and target three-dimensional point cloud data.

[0013] In a second aspect, an embodiment of the present application provides a greenhouse tomato plant growth parameter monitoring device, comprising: An initial multimodal data acquisition module is used to: acquire initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple viewing angles; the initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; A target multimodal data acquisition module is used to: pre-process the initial multimodal data to obtain target multimodal data; A three-dimensional reconstruction data acquisition module, used for fusion modeling of the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; A key growth part identification module, used for identifying the key growth part of the tomato plant based on the three-dimensional reconstruction data; A target growth parameter acquisition module, used to obtain the target growth parameters of the key growth part based on the three-dimensional reconstruction data corresponding to the key growth part; The target growth parameter prediction module is used to obtain the future prediction value of the target growth parameter based on the current time series data of the target growth parameter.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a computer program, wherein when the processor executes the program, the steps of the method for monitoring growth parameters of greenhouse tomato plants described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the greenhouse tomato plant growth parameter monitoring method described in the first aspect.

[0016] The greenhouse tomato plant growth parameter monitoring method, device, equipment and product provided in the present application obtain initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple perspectives, the initial multimodal data including initial RGB image data, initial depth image data and initial three-dimensional point cloud data, pre-process the initial multimodal data to obtain target multimodal data, perform fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant, identify key growth parts of the tomato plant based on the three-dimensional reconstruction data, obtain target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts, and obtain future predicted values ​​of the target growth parameters based on current time series data of the target growth parameters. The present application acquires RGB image data, depth image data and three-dimensional point cloud data of tomato plants under multiple perspectives, thereby increasing the data acquisition dimension on the one hand, and on the other hand, through multi-perspective acquisition, avoiding the influence of overlapping occlusion between plants on the integrity of data acquisition. After three-dimensional reconstruction of the plant based on these data, the key parts of the plant can be accurately identified and its target growth parameters can be obtained. By predicting the future value of the target growth parameter through the current time series data of the target growth parameter, the dynamic growth data of the target growth parameter can be obtained, thereby realizing dynamic monitoring of the target growth parameters of the plant. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow diagrams of the method for monitoring growth parameters of greenhouse tomato plants provided in the embodiments of the present application; Figure 2 This is the second flow chart of the method for monitoring growth parameters of greenhouse tomato plants provided in the embodiment of the present application; Figure 3 This is the third flow chart of the method for monitoring growth parameters of greenhouse tomato plants provided in the embodiment of the present application; Figure 4 This is the fourth flow chart of the method for monitoring growth parameters of greenhouse tomato plants provided in the embodiment of the present application; Figure 5 This is the fifth flow chart of the method for monitoring growth parameters of greenhouse tomato plants provided in the embodiment of the present application; Figure 6 It is a schematic diagram of the structure of a greenhouse tomato plant growth parameter monitoring device provided in an embodiment of the present application; Figure 7It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] It should be noted that in the description of the embodiments of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0021] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] Figure 1 This is one of the flow charts of the method for monitoring greenhouse tomato plant growth parameters provided in the embodiment of the present application. Figure 1 The present application embodiment provides a method for monitoring growth parameters of greenhouse tomato plants, which may include: 101. Obtain initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse from multiple perspectives; The initial multimodal data includes initial RGB image data, initial depth image data and initial three-dimensional point cloud data; 102. Preprocessing the initial multimodal data to obtain target multimodal data; 103. Perform fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; 104. Identify key growth parts of tomato plants based on 3D reconstruction data; 105. Based on the three-dimensional reconstruction data corresponding to the key growth part, obtain the target growth parameter of the key growth part; 106. Based on the current time series data of the target growth parameter, obtain a future predicted value of the target growth parameter.

[0023] In step 101, an RGB camera, a depth camera, and a LiDAR (Light Detection and Ranging) sensor are installed on the collection vehicle. When the collection vehicle moves in the target greenhouse, the RGB camera, the depth camera, and the LiDAR sensor respectively collect initial RGB image data, initial depth image data, and initial three-dimensional point cloud data of tomato plants from multiple perspectives at a high frequency.

[0024] The RGB camera has a resolution of 4096 pixels × 3000 pixels, a pixel size of 3.45 microns, and supports full-resolution output at up to 60 frames per second, which can capture the detailed texture of tomato plants. A 12mm fixed-focus lens is used with a field of view of approximately 90 degrees to obtain a larger single-frame coverage.

[0025] The resolution of the depth camera is 1024 pixels × 1024 pixels, the depth measurement range is 0.25 meters to 2.88 meters, the system error is less than 11 mm at a distance of 0.5 meters, and less than 18 mm at a distance of 2 meters. The system error increases linearly with the increase of distance. The frame rate is 30 frames per second, and it is triggered synchronously with the RGB camera.

[0026] The LiDAR sensor uses a 16-beam laser, with a vertical angular resolution of 2 degrees, a horizontal field of view of 360 degrees, a vertical field of view of 30 degrees, an error of ±15 degrees, a ranging range of 100 meters, an accuracy of ±3 centimeters, and emits 300,000 laser pulses per second to generate precise three-dimensional point clouds.

[0027] The pulse output signal of the LiDAR sensor is connected to the external trigger input ports of the RGB camera and the depth camera respectively. The LiDAR sensor outputs a synchronization signal every time it rotates one circle. The two cameras are exposed and record the trigger timestamp at the same time. The timestamp accuracy reaches the microsecond level, realizing the synchronous triggering of the two cameras.

[0028] In step 102, after preprocessing the initial RGB image data, the initial depth image data and the initial three-dimensional point cloud data, target RGB image data, target depth image data and target three-dimensional point cloud data are obtained accordingly.

[0029] The greenhouse tomato plant growth parameter monitoring method provided in the present embodiment obtains initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple perspectives, the initial multimodal data including initial RGB image data, initial depth image data and initial three-dimensional point cloud data, pre-processes the initial multimodal data to obtain target multimodal data, performs fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plants, identifies key growth parts of the tomato plants based on the three-dimensional reconstruction data, obtains target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts, and obtains future prediction values ​​of the target growth parameters based on current time series data of the target growth parameters. This embodiment acquires RGB image data, depth image data and three-dimensional point cloud data of tomato plants under multiple perspectives, which, on the one hand, increases the data acquisition dimension; on the other hand, through multi-perspective acquisition, it can avoid the influence of overlapping occlusion between plants on the integrity of data acquisition. After three-dimensional reconstruction of the plant based on these data, the key parts of the plant can be accurately identified and its target growth parameters can be obtained. The future value of the target growth parameter can be predicted by the current time series data of the target growth parameter, and the dynamic growth data of the target growth parameter can be obtained, thereby realizing dynamic monitoring of the target growth parameters of the plant.

[0030] Figure 2 This is the second flow chart of the method for monitoring greenhouse tomato plant growth parameters provided in the embodiment of the present application. Figure 2 In one embodiment, the initial multimodal data under multiple viewing angles may be collected based on the following methods: 201. Construct a target path function of the collection vehicle based on the positions of the plurality of tomato plants in the target greenhouse, the adjacent collection positions of the plurality of tomato plants, the position of the collection vehicle, and the speed of the collection vehicle; The speed includes linear speed and angular speed; 202. Adjust the linear velocity and angular velocity of the collection vehicle until the value of the target path function is minimized, and determine the linear velocity and angular velocity of the collection vehicle at this time as the optimal linear velocity and optimal angular velocity; 203. Based on the optimal angular velocity, the optimal heading angle of the collection vehicle is obtained; 204. Based on the optimal linear speed and the optimal heading angle, control the movement of the collection vehicle and collect multimodal data of each tomato plant in the target greenhouse to obtain initial multimodal data under multiple perspectives.

[0031] From step 201 to step 202, the minimum value of the target path function of the collection vehicle can be expressed as: ; in, To collect the position of the car, for Tomato plant The location of the tomato plants, To collect the linear speed of the car, For the A set of serial indexes of the collected locations near the tomato plant, for Internal serial number The corresponding collected locations, and is the balance factor, for The derivative with respect to time, for The derivative with respect to time, To collect the heading angle of the car, for The derivative with respect to time, is the angular velocity of the collected car, It is a collection of the spatial ranges within which the car can travel in the target greenhouse.

[0032] Based on the above formula, the optimal line speed can be obtained and the optimal angular velocity .

[0033] In step 203, Substitute into the formula based on , we can get the optimal heading angle .

[0034] In step 204, based on and , control the movement of the collection trolley and collect multimodal data of each tomato plant in the target greenhouse. It should be noted that during the movement of the collection trolley, and The data will also change in real time, so that the collection vehicle can always collect data of tomato plants based on the current optimal linear speed and heading angle, and obtain initial multimodal data from multiple perspectives.

[0035] This embodiment not only makes the data collection of the collection vehicle cover each tomato plant by planning the driving route of the collection vehicle, but also encourages the collection vehicle to collect a variety of initial multimodal data from multiple perspectives near each plant by adding the distance between the position of the collection vehicle and the position of the collected data adjacent to each tomato plant as a penalty item in the target path function, thereby improving data diversity.

[0036] Figure 3 This is the third flow chart of the method for monitoring greenhouse tomato plant growth parameters provided in the embodiment of the present application. Figure 3 In one embodiment, fusion modeling is performed on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant, which may include: 301. Inputting the target multimodal data into a pre-trained semantic segmentation network to obtain target multimodal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network; The target multimodal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data and target 3D point cloud semantic segmentation data; 302. Calculate the attention weights of the target multimodal semantic segmentation data of different parts in the feature dimension and the spatial dimension, and obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data; 303. Obtaining texture features of the target RGB image data based on the first feature attention weight and the first spatial attention weight; 304. Obtaining a depth feature of the target depth image data based on the second feature attention weight and the second spatial attention weight; 305. Obtain a normal vector feature of the target three-dimensional point cloud data based on the third feature attention weight and the third spatial attention weight; 306. Calculate the attention weight between each two features among the texture features, the depth features, and the normal vector features to obtain a first association weight between the texture feature and the depth feature, a second association weight between the texture feature and the normal vector feature, and a third association weight between the depth feature and the normal vector feature; 307. Based on the first association weight and the second association weight, obtain a first fusion feature with the texture feature as the central feature; 308. Based on the first association weight and the third association weight, obtain a second fusion feature with the depth feature as the central feature; 309. Based on the second association weight and the third association weight, a third fusion feature with the normal vector feature as the central feature is obtained; 310. Perform weighted summation on the first fusion feature, the second fusion feature, and the third fusion feature to obtain three-dimensional reconstructed data of the tomato plant.

[0037] In step 301, different parts including leaves, stems, flowers, etc. of the tomato plant are processed by the semantic segmentation network, and the output target RGB image semantic segmentation data, target depth image semantic segmentation data and target three-dimensional point cloud semantic segmentation data of each part all contain their own contextual semantic information.

[0038] In step 302, the attention weights of the target RGB image semantic segmentation data of different parts in the feature dimension and the spatial dimension are calculated to obtain the first feature attention weight and the first spatial attention weight, the attention weights of the target depth image semantic segmentation data of different parts in the feature dimension and the spatial dimension are calculated to obtain the second feature attention weight and the second spatial attention weight, and the attention weights of the target three-dimensional point cloud semantic segmentation data in the feature dimension and the spatial dimension are calculated to obtain the third feature attention weight and the third spatial attention weight.

[0039] In step 303 to step 305, the corresponding features of the target RGB image data, the target depth image data and the target three-dimensional point cloud data are respectively extracted using the self-attention mechanism.

[0040] From step 306 to step 310, the features obtained by the self-attention mechanism are fused layer by layer using the cross-attention mechanism to obtain the final three-dimensional reconstruction data of the tomato plant.

[0041] In the above steps, the 3D reconstruction data of the tomato plant must satisfy the constraint of minimizing the first total loss function. It can be expressed as: ; in, is the photometric consistency loss, which is used to evaluate the photometric loss of multi-view data. is the semantic consistency loss, which is used to evaluate the semantic loss of multi-view data. is the multi-view adversarial loss, which is used to evaluate the adversarial loss of multi-view data. is the temporal consistency loss, which is used to evaluate the temporal loss of multi-view data. , , and is the weight of each loss.

[0042] In this embodiment, the target multimodal data is first input into the semantic segmentation network so that the output target multimodal semantic segmentation data has contextual semantic information, and then the self-attention weights of the target multimodal semantic segmentation data of different parts in the feature dimension and the spatial dimension are calculated. Based on the self-attention mechanism, the corresponding features of the target RGB image data, the target depth image data and the target three-dimensional point cloud data are respectively calculated. Finally, the features are fused layer by layer using the cross-attention mechanism to obtain the three-dimensional reconstruction data of the tomato plant. Under the constraints of multiple losses, the multimodal data of multiple perspectives are fully utilized to complement each other from three aspects: semantic segmentation, self-attention mechanism and cross-attention mechanism, so as to obtain accurate three-dimensional reconstruction data of the tomato plant to the maximum extent.

[0043] Figure 4 This is the fourth flow chart of the method for monitoring greenhouse tomato plant growth parameters provided in the embodiment of the present application. Figure 4 In one embodiment, identifying key growth parts of a tomato plant based on three-dimensional reconstruction data may include: 401. Input the three-dimensional reconstruction data into the Swin Transformer network to obtain three-dimensional reconstruction features of multiple stages in the Swin Transformer network; 402. Aggregate the three-dimensional reconstruction features of adjacent stages to obtain a plurality of first aggregated features; 403. Group the multiple first aggregate features to obtain multiple groups of first aggregate features; 404. Calculate the attention weight of each group of first aggregated features, and aggregate multiple groups of first aggregated features based on the attention weights to obtain second aggregated features; 405. Identify key growth parts of the tomato plant based on the second aggregated feature.

[0044] In step 402, the aggregation of the 3D reconstruction features of adjacent stages may be performed according to the following formula: ; in, For the The first fusion feature of the stage, For the The three-dimensional reconstruction features of each stage, For the The first fusion feature of the stage, represents the upsampling operation, It is a 1×1 convolutional layer.

[0045] This aggregation process enhances the information interaction between high-level semantic features and low-level detail features in 3D reconstruction features, making the aggregated feature representation richer and more recognizable.

[0046] In step 403 to step 404, after the multiple first aggregate features are grouped, the self-attention weight of each group is calculated based on the first aggregate features in each group, and then the self-attention weight of each group is used to perform weighted summation on the first aggregate features of all groups, thereby obtaining a second aggregate feature with global semantic information.

[0047] This method significantly reduces the computational complexity of global attention through group calculation, enabling it to be efficiently applied to large-scale feature maps and retain more spatial details while capturing global semantic information.

[0048] In step 405, after two aggregations, sufficiently fine features of each part of the tomato plant are obtained, so that each key growth part can be accurately identified.

[0049] In the above steps, identifying the key growth parts of tomato plants needs to satisfy the constraint of minimizing the second total loss function. It can be expressed as: ; in, It is the information noise contrast estimation loss, which is used to evaluate the discriminability between foreground data and background data in multi-view data. The smaller the loss, the stronger the discriminability. is the relative position loss, which is used to evaluate the loss between the predicted distance and the actual distance between the center points of any two parts of the tomato plant in the multi-view data. is the size ratio loss, which is used to evaluate the loss between the predicted size and the actual size of any part of the tomato plant in the multi-view data. , and is the weight of each loss.

[0050] In view of the characteristics of dense distribution of tomato plants and complex and changeable background in the greenhouse, this embodiment first uses a Swin Transformer (Shifted Window Transformer) network to extract 3D reconstruction features of multiple stages from the 3D reconstruction data under the constraints of foreground and background discriminability and prior knowledge of distances and sizes of tomato plant parts, and then aggregates the 3D reconstruction features of adjacent stages based on upsampling and convolutional layers to extract richer and more recognizable first aggregate features, and then groups the multiple first aggregate features, obtains the self-attention weight of each group, and then re-aggregates the first aggregate features of each group to extract second aggregate features with global semantic information. At this time, the features of each part of the tomato plant are sufficiently fine and obvious, so that its key growth parts can be accurately identified.

[0051] In one embodiment, obtaining target growth parameters of the key growth part based on the three-dimensional reconstruction data corresponding to the key growth part may include: Inputting the three-dimensional reconstruction data corresponding to the key growth part into the target growth parameter identification model to obtain the target growth parameter output by the target growth parameter identification model; The key growth parts include the flower of the tomato plant, the leaf of the tomato plant, the stem of the tomato plant and the apex of the tomato plant, and the target growth parameters include the growth period category of the flower, the length of the leaf, the stem diameter of the stem and the height of the apex; The target growth parameter recognition model is obtained by training a basic model that matches the target growth parameter recognition through historical three-dimensional reconstruction data of key growth parts and their corresponding target growth parameter labels.

[0052] Specifically, the target growth parameter recognition model may include a growth period category recognition model, a length recognition model, a stem diameter recognition model, and a height recognition model, wherein: 1. The growth period category recognition model can be expressed as follows: ; in, The growth period of flowers, including bud stage, flowering stage and withering stage. is the activation function, is the growth period weight matrix of the fully connected layer. By introducing this weight matrix, different weights can be assigned to flowers in different growth periods, thereby more accurately describing the dynamic process of the flower life cycle. is the maximum pooling operation, is a three-dimensional convolutional layer, The three-dimensional reconstruction data corresponding to the flower.

[0053] The historical 3D reconstruction data of flowers used to train the above model needs to cover different growth stages of flowers so that the model can recognize different growth period categories of flowers.

[0054] 2. The length recognition model can be expressed as follows: ; in, is the length of the blade, is the attention pooling layer, For multi-scale grouping operations, Backbone network for point cloud data processing, It is the 3D reconstruction data corresponding to the blade.

[0055] This model does not need to explicitly extract the leaf skeleton and fit the vein curve, and can realize end-to-end recognition of leaf length, greatly improving the robustness and efficiency of recognition.

[0056] 3. The stem thickness recognition model can be expressed as follows: ; ; ; in, The main stem The cross section is the center, The 3D reconstruction data set in the local neighborhood with the viewing angle as the direction, is the data coordinate in the collection, For The radius of the circle to be fitted, For The coordinates of the center of the circle for circle fitting, are the data coordinates and radius obtained after circle fitting, The main stem cross section, The stem thickness from each perspective, for The weight of It is the weighted average of the stem diameters at all cross sections and all viewing angles of the main stem, which is also the stem diameter finally output by the model.

[0057] It should be noted that all cross sections of the main stem were measured at equal intervals in the direction of the main stem. The measurement quality can be adaptively determined based on the viewing angle and cross section, and the measurement quality can be evaluated by factors such as fitting residuals and data density.

[0058] The model integrates stem thickness estimation from multiple cross sections and multiple perspectives, effectively reducing the impact of local occlusion and noise, and improving the accuracy and stability of stem thickness estimation.

[0059] 4. The highly identifiable model can be expressed as follows: ; in, For the vertex The height of the moment, Three-dimensional reconstruction data for vertices The maximum height of is the maximum height relative growth rate, It is the moment when the maximum height relative to the growth rate is the largest.

[0060] This embodiment uses differentiated target growth parameter recognition models to identify target growth parameters for different key growth parts: for the flowers of tomato plants, a lightweight convolutional neural network is used to identify the growth period category of the flowers through three-dimensional feature extraction and fully connected layer mapping with the three-dimensional reconstruction data of the flowers as input; for the leaves of tomato plants, a PointNet++ The backbone network uses the three-dimensional reconstruction data of the leaves as input, and identifies the length of the leaves through multi-scale feature learning and attention mechanism; for the stems of the tomato plants, a circle fitting method is used, and the three-dimensional reconstruction data of the main stem with multiple cross-sections and multiple perspectives is used as input, and the stem thickness of the stem is identified by fusing the stem thickness estimates of multiple cross-sections and multiple perspectives; for the apex of the tomato plant, a growth curve fitting method is used, and the three-dimensional reconstruction data of the apex is used as input, and the height of the apex is identified by fitting the vertex growth curve. This embodiment uses different methods to fully mine the three-dimensional features from the three-dimensional reconstruction data of different key growth parts according to their characteristics, so as to achieve accurate identification of their target growth parameters.

[0061] Figure 5 This is the fifth flow chart of the method for monitoring greenhouse tomato plant growth parameters provided in the embodiment of the present application. Figure 5 In one embodiment, obtaining a future predicted value of a target growth parameter based on current time series data of the target growth parameter may include: 501. Generate lagged time series data and moving average time series data of the target growth parameter based on the current time series data of the target growth parameter; 502. Splicing the current time series data, the lagged time series data, and the moving average time series data to obtain spliced ​​time series data; 503. Convolution and pooling are performed on the spliced ​​time series data to obtain multiple local features of the target growth parameters in the temporal and spatial dimensions; 504. Capture the long-term dependency relationship between each local feature and the local features before and after it in time sequence, and obtain multiple long-term dependent local features of the target growth parameter in the spatiotemporal dimension; 505. Calculate the attention weights between multiple long-term dependent local features; 506. Based on the attention weight, the hidden state representation of the target growth parameter in the spatiotemporal dimension is calculated with each long-term dependent local feature as the center to obtain multiple hidden state representations; 507. Map multiple hidden state representations to the output space to obtain future predicted values ​​of the target growth parameters.

[0062] The aforementioned length recognition model, stem thickness recognition model and height recognition model can generally only identify the current target growth parameters based on the current three-dimensional reconstruction data corresponding to the input key growth parts, but cannot predict the future target growth parameters. In this embodiment, the time series data obtained by multiple identifications can be used to realize the prediction of future target growth parameters.

[0063] In step 502 to step 503, for any target growth parameter, the current time series data can be calculated according to the following formula: , delayed time series data and moving average time series data Perform concatenation, convolution and pooling to obtain a set of multiple local features of the target growth parameters in the spatiotemporal dimension. : ; in, is a one-dimensional maximum pooling operation, is the activation function, is a one-dimensional convolutional layer, It includes local features of multiple time steps.

[0064] In step 504, for In the time step Local features , the long-term dependency between it and the local features of the previous and next time series can be captured using the following formula: ; ; in, is the time step Local features The long-term dependency between the local features and their predecessors is expressed as is the time step Local features and The long-term dependency between is the time step Local features The long-term dependency between the local features and their successors is expressed as is the time step Local features and The long-term dependency between It is a bidirectional gated recurrent unit (BiGRU) network.

[0065] Then, the long-term dependent local characteristics of the target growth parameter in the spatiotemporal dimension are obtained according to the following formula: : ; In step 506, Taking it as an example, the attention weights between it and other long-term dependent local features are used to perform weighted summation on other long-term dependent local features to obtain a hidden state representation of the target growth parameter in the spatiotemporal dimension. Similarly, other long-term dependent local features can be taken as examples to obtain other hidden state representations of the target growth parameter in the spatiotemporal dimension.

[0066] In step 507, these hidden state representations are mapped to the output space through a fully connected layer to obtain the future predicted values ​​of the target growth parameters.

[0067] In the above steps, the future predicted value of the target growth parameter must satisfy the constraint of minimizing the asymmetric loss function. The asymmetric loss function can be expressed as : ; in, For future predictions, is the future actual value, and They are the penalty coefficients for overestimation and underestimation, respectively, which can be set according to actual needs and are not limited here.

[0068] This embodiment first splices multiple time series data of the target growth parameter to obtain spliced ​​time series data, and then uses the one-dimensional convolution and one-dimensional pooling operations in the convolutional neural network to automatically learn the high-order patterns in the spliced ​​time series data to extract the local features of the target growth parameter in the spatiotemporal dimension, and then uses the bidirectional gated recurrent unit network to learn the long-term dependencies between adjacent time series local features, and then calculates multiple hidden state representations of the target growth parameter in the spatiotemporal dimension according to the attention weight, and then maps them to the output space to obtain the future prediction value of the target growth parameter. Through the hybrid architecture of the convolutional neural network and the bidirectional gated recurrent unit network, the hierarchical feature learning is used to enhance the representation ability of complex spliced ​​time series data, so that accurate future prediction values ​​of the target growth parameters can be obtained.

[0069] In one embodiment, preprocessing the initial multimodal data to obtain target multimodal data may include: 1. Perform exposure correction, image enhancement, distortion correction and denoising on the initial RGB image data in sequence to obtain the RGB image data to be processed. Specifically: 1.1. Using the automatic exposure algorithm of the RGB camera to correct the exposure time of the initial RGB image data, to obtain the first corrected RGB image data; 1.2. Perform pixel-level image enhancement on the first corrected RGB image data using an adaptive histogram equalization algorithm to obtain second corrected RGB image data; The adaptive histogram equalization algorithm performs block processing on the first corrected RGB image data and performs histogram equalization on the first corrected RGB image data of each local area, thereby improving the contrast of the image data and maintaining the local details. 1.3. Perform pixel-level lens distortion correction on the second corrected RGB image data using a wide-angle Brown-Conrady distortion model to obtain third corrected RGB image data; 1.4. De-noise the third corrected RGB image data using guided filtering, smooth the image data while maintaining image edge data, and obtain RGB image data to be processed.

[0070] 2. Fill the initial depth image data with invalid values ​​and perform denoising in sequence to obtain the depth image data to be processed; 3. Perform outlier removal, downsampling and normal vector estimation on the initial 3D point cloud data in turn to obtain the 3D point cloud data to be processed; After the above preprocessing is completed, the RGB image data to be processed, the depth image data to be processed and the 3D point cloud data to be processed are aligned to obtain the target RGB image data, the target depth image data and the target 3D point cloud data. Specifically: The Zhang Zhengyou calibration method is used to estimate the RGB camera intrinsic parameters and the depth camera intrinsic parameters. The RGB camera intrinsic parameters are used to transform the RGB image data to be processed into the RGB camera coordinate system. The depth camera intrinsic parameters are used to transform the depth image data to be processed into the depth camera coordinate system. The black and white checkerboard plane of the RGB camera coordinate system and the black and white checkerboard plane of the LiDAR sensor coordinate system are registered using the Iterative Closest Point (ICP) algorithm, as well as the black and white checkerboard plane of the depth camera coordinate system and the black and white checkerboard plane of the LiDAR sensor coordinate system, so as to establish the transformation extrinsics between the two camera coordinate systems and the LiDAR sensor coordinate system respectively; Each frame of 3D point cloud data to be processed is transformed from the LiDAR sensor coordinate system to the RGB camera coordinate system and the depth camera coordinate system based on the following formula: ; in, is the pixel coordinate of the 3D point cloud data to be processed in the RGB camera coordinate system or the depth camera coordinate system. is the corresponding transformation extrinsic parameter, It is the homogeneous coordinate of the 3D point cloud data to be processed in the LiDAR sensor coordinate system.

[0071] This embodiment can improve data quality by performing targeted preprocessing on initial RGB image data, initial depth image data, and initial three-dimensional point cloud data, and performing data alignment, which is helpful for the effective fusion of subsequent target multimodal data.

[0072] The greenhouse tomato plant growth parameter monitoring device provided in an embodiment of the present application is described below. The greenhouse tomato plant growth parameter monitoring device described below and the greenhouse tomato plant growth parameter monitoring method described above can be referenced to each other.

[0073] Figure 6 Schematic diagram of the structure of the greenhouse tomato plant growth parameter monitoring device provided in the embodiment of the present application. Figure 6 The present application embodiment provides a greenhouse tomato plant growth parameter monitoring device, which may include: The initial multimodal data acquisition module 601 is used to: acquire the initial multimodal data of the tomato plants collected by the collection vehicle in the target greenhouse under multiple viewing angles; the initial multimodal data includes initial RGB image data, initial depth image data and initial three-dimensional point cloud data; The target multimodal data acquisition module 602 is used to: pre-process the initial multimodal data to obtain target multimodal data; A three-dimensional reconstruction data acquisition module 603 is used to perform fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; A key growth part identification module 604, configured to identify the key growth part of the tomato plant based on the three-dimensional reconstruction data; A target growth parameter acquisition module 605 is used to obtain the target growth parameter of the key growth part based on the three-dimensional reconstruction data corresponding to the key growth part; The target growth parameter prediction module 606 is used to obtain a future prediction value of the target growth parameter based on the current time series data of the target growth parameter.

[0074] The greenhouse tomato plant growth parameter monitoring device provided in the present embodiment obtains initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple perspectives, the initial multimodal data including initial RGB image data, initial depth image data and initial three-dimensional point cloud data, pre-processes the initial multimodal data to obtain target multimodal data, performs fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plants, identifies key growth parts of the tomato plants based on the three-dimensional reconstruction data, obtains target growth parameters of the key growth parts based on the three-dimensional reconstruction data corresponding to the key growth parts, and obtains future prediction values ​​of the target growth parameters based on current time series data of the target growth parameters. This embodiment acquires RGB image data, depth image data and three-dimensional point cloud data of tomato plants under multiple perspectives, which, on the one hand, increases the data acquisition dimension; on the other hand, through multi-perspective acquisition, it can avoid the influence of overlapping occlusion between plants on the integrity of data acquisition. After three-dimensional reconstruction of the plant based on these data, the key parts of the plant can be accurately identified and its target growth parameters can be obtained. The future value of the target growth parameter can be predicted by the current time series data of the target growth parameter, and the dynamic growth data of the target growth parameter can be obtained, thereby realizing dynamic monitoring of the target growth parameters of the plant.

[0075] In one embodiment, an initial multimodal data acquisition module (not shown in the figure) is further included, which is used to: Based on the positions of the plurality of tomato plants in the target greenhouse, the adjacent collected positions of the plurality of tomato plants, the position of the collection vehicle and the speed of the collection vehicle, a target path function of the collection vehicle is constructed; the speed includes a linear speed and an angular speed; Adjusting the linear speed and angular speed of the collection vehicle until the value of the target path function is minimized, and determining the linear speed and angular speed of the collection vehicle at this time as the optimal linear speed and optimal angular speed; Based on the optimal angular velocity, an optimal heading angle of the collection vehicle is obtained; Based on the optimal linear speed and the optimal heading angle, the collection vehicle is controlled to move and collect multimodal data of each tomato plant in the target greenhouse to obtain initial multimodal data under the multiple viewing angles.

[0076] In one embodiment, the 3D reconstruction data acquisition module 603 is specifically used to: Inputting the target multimodal data into a pre-trained semantic segmentation network to obtain target multimodal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network; the target multimodal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data and target three-dimensional point cloud semantic segmentation data; Calculate the attention weights of the target multimodal semantic segmentation data of the different parts in the feature dimension and the spatial dimension, and obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data; Obtaining texture features of target RGB image data based on the first feature attention weight and the first spatial attention weight; Obtaining a depth feature of the target depth image data based on the second feature attention weight and the second spatial attention weight; Based on the third feature attention weight and the third spatial attention weight, obtaining a normal vector feature of the target three-dimensional point cloud data; Calculating the attention weight between each two features among the texture feature, the depth feature and the normal vector feature to obtain a first association weight between the texture feature and the depth feature, a second association weight between the texture feature and the normal vector feature, and a third association weight between the depth feature and the normal vector feature; Based on the first association weight and the second association weight, obtaining a first fusion feature with the texture feature as a central feature; Based on the first association weight and the third association weight, obtaining a second fusion feature with the depth feature as a central feature; Based on the second association weight and the third association weight, obtaining a third fusion feature with the normal vector feature as a central feature; The first fusion feature, the second fusion feature and the third fusion feature are weightedly summed to obtain the three-dimensional reconstruction data of the tomato plant.

[0077] In one embodiment, the key growth site identification module 604 is specifically configured to: Inputting the three-dimensional reconstruction data into a Swin Transformer network to obtain three-dimensional reconstruction features of multiple stages in the Swin Transformer network; Aggregating the three-dimensional reconstruction features of adjacent stages to obtain a plurality of first aggregated features; Grouping the multiple first aggregate features to obtain multiple groups of first aggregate features; Calculating the attention weight of each group of first aggregate features, and aggregating the multiple groups of first aggregate features based on the attention weight to obtain a second aggregate feature; Key growth parts of the tomato plant are identified based on the second aggregated feature.

[0078] In one embodiment, the target growth parameter acquisition module 605 is specifically used to: Inputting the three-dimensional reconstruction data corresponding to the key growth part into a target growth parameter recognition model to obtain the target growth parameter output by the target growth parameter recognition model; The key growth parts include the flower of the tomato plant, the leaves of the tomato plant, the stem of the tomato plant and the apex of the tomato plant, and the target growth parameters include the growth period category of the flower, the length of the leaves, the stem thickness of the stem and the height of the apex; The target growth parameter identification model is obtained by training a basic model that matches the target growth parameter identification through the historical three-dimensional reconstruction data of the key growth part and its corresponding target growth parameter label.

[0079] In one embodiment, the target growth parameter prediction module 606 is specifically used to: Based on the current time series data of the target growth parameter, generating the lagged time series data and the moving average time series data of the target growth parameter; splicing the current time series data, the lagged time series data and the moving average time series data to obtain spliced ​​time series data; Convolution and pooling are performed on the spliced ​​time series data to obtain multiple local features of the target growth parameter in the spatiotemporal dimension; Capturing the long-term dependency relationship between each local feature and the local features before and after it, and obtaining a plurality of long-term dependent local features of the target growth parameter in the spatiotemporal dimension; Calculate the attention weights between multiple long-term dependent local features; Based on the attention weights, a hidden state representation of the target growth parameter in the spatiotemporal dimension is calculated with each long-term dependent local feature as the center to obtain multiple hidden state representations; The multiple hidden state representations are mapped to an output space to obtain future predicted values ​​of the target growth parameters.

[0080] In one embodiment, the target multimodal data acquisition module 602 is specifically used to: Performing exposure correction, image enhancement, distortion correction and denoising on the initial RGB image data in sequence to obtain RGB image data to be processed; Performing invalid value filling and denoising on the initial depth image data in sequence to obtain depth image data to be processed; Performing outlier removal, downsampling, and normal vector estimation on the initial three-dimensional point cloud data in sequence to obtain three-dimensional point cloud data to be processed; The RGB image data to be processed, the depth image data to be processed and the three-dimensional point cloud data to be processed are aligned to obtain target RGB image data, target depth image data and target three-dimensional point cloud data.

[0081] FIG. 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call a computer program in the memory 730 to execute the steps of the method for monitoring growth parameters of greenhouse tomato plants, for example, including: Acquire initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple viewing angles; the initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; Preprocessing the initial multimodal data to obtain target multimodal data; Performing fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; identifying key growth parts of the tomato plant based on the three-dimensional reconstruction data; Based on the three-dimensional reconstruction data corresponding to the key growth part, a target growth parameter of the key growth part is obtained; Based on the current time series data of the target growth parameter, a future predicted value of the target growth parameter is obtained.

[0082] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] On the other hand, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can perform the steps of the greenhouse tomato plant growth parameter monitoring method provided in the above embodiments, for example, including: Acquire initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple viewing angles; the initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; Preprocessing the initial multimodal data to obtain target multimodal data; Performing fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; identifying key growth parts of the tomato plant based on the three-dimensional reconstruction data; Based on the three-dimensional reconstruction data corresponding to the key growth part, a target growth parameter of the key growth part is obtained; Based on the current time series data of the target growth parameter, a future predicted value of the target growth parameter is obtained.

[0084] On the other hand, the embodiments of the present application further provide a non-transitory computer-readable storage medium on which a computer program is stored, wherein the computer program is used to enable a processor to execute the steps of the greenhouse tomato plant growth parameter monitoring method provided in the above embodiments, for example, including: Acquire initial multimodal data of tomato plants collected by a collection vehicle in a target greenhouse under multiple viewing angles; the initial multimodal data includes initial RGB image data, initial depth image data, and initial three-dimensional point cloud data; Preprocessing the initial multimodal data to obtain target multimodal data; Performing fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; identifying key growth parts of the tomato plant based on the three-dimensional reconstruction data; Based on the three-dimensional reconstruction data corresponding to the key growth part, a target growth parameter of the key growth part is obtained; Based on the current time series data of the target growth parameter, a future predicted value of the target growth parameter is obtained.

[0085] The non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for monitoring growth parameters of greenhouse tomato plants, characterized in that: include: Obtaining initial multimodal data of tomato plants collected by the collection vehicle in the target greenhouse from multiple perspectives; The initial multimodal data includes initial RGB image data, initial depth image data and initial three-dimensional point cloud data; Preprocessing the initial multimodal data to obtain target multimodal data; Performing fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; identifying key growth parts of the tomato plant based on the three-dimensional reconstruction data; Based on the three-dimensional reconstruction data corresponding to the key growth part, a target growth parameter of the key growth part is obtained; Based on the current time series data of the target growth parameter, a future predicted value of the target growth parameter is obtained.

2. The method for monitoring growth parameters of greenhouse tomato plants according to claim 1, characterized in that: The initial multimodal data under multiple perspectives is collected based on the following methods: Based on the positions of the plurality of tomato plants in the target greenhouse, the adjacent collected positions of the plurality of tomato plants, the position of the collection vehicle and the speed of the collection vehicle, a target path function of the collection vehicle is constructed; the speed includes a linear speed and an angular speed; Adjusting the linear speed and angular speed of the collection vehicle until the value of the target path function is minimized, and determining the linear speed and angular speed of the collection vehicle at this time as the optimal linear speed and optimal angular speed; Based on the optimal angular velocity, an optimal heading angle of the collection vehicle is obtained; Based on the optimal linear speed and the optimal heading angle, the collection vehicle is controlled to move and collect multimodal data of each tomato plant in the target greenhouse to obtain initial multimodal data under the multiple viewing angles.

3. The method for monitoring growth parameters of greenhouse tomato plants according to claim 1, characterized in that: The fusion modeling of the target multimodal data to obtain the three-dimensional reconstruction data of the tomato plant includes: Inputting the target multimodal data into a pre-trained semantic segmentation network to obtain target multimodal semantic segmentation data of different parts of the tomato plant output by the pre-trained semantic segmentation network; the target multimodal semantic segmentation data includes target RGB image semantic segmentation data, target depth image semantic segmentation data and target three-dimensional point cloud semantic segmentation data; Calculate the attention weights of the target multimodal semantic segmentation data of the different parts in the feature dimension and the spatial dimension, and obtain the first feature attention weight and the first spatial attention weight corresponding to the target RGB image semantic segmentation data, the second feature attention weight and the second spatial attention weight corresponding to the target depth image semantic segmentation data, and the third feature attention weight and the third spatial attention weight corresponding to the target three-dimensional point cloud semantic segmentation data; Obtaining texture features of target RGB image data based on the first feature attention weight and the first spatial attention weight; Obtaining a depth feature of the target depth image data based on the second feature attention weight and the second spatial attention weight; Based on the third feature attention weight and the third spatial attention weight, obtaining a normal vector feature of the target three-dimensional point cloud data; Calculating the attention weight between each two features among the texture feature, the depth feature and the normal vector feature to obtain a first association weight between the texture feature and the depth feature, a second association weight between the texture feature and the normal vector feature, and a third association weight between the depth feature and the normal vector feature; Based on the first association weight and the second association weight, obtaining a first fusion feature with the texture feature as a central feature; Based on the first association weight and the third association weight, obtaining a second fusion feature with the depth feature as a central feature; Based on the second association weight and the third association weight, obtaining a third fusion feature with the normal vector feature as a central feature; The first fusion feature, the second fusion feature and the third fusion feature are weightedly summed to obtain the three-dimensional reconstruction data of the tomato plant.

4. The method for monitoring growth parameters of greenhouse tomato plants according to claim 1, characterized in that: The identifying the key growth part of the tomato plant based on the three-dimensional reconstruction data comprises: Inputting the three-dimensional reconstruction data into a Swin Transformer network to obtain three-dimensional reconstruction features of multiple stages in the Swin Transformer network; Aggregating the three-dimensional reconstruction features of adjacent stages to obtain a plurality of first aggregated features; Grouping the multiple first aggregate features to obtain multiple groups of first aggregate features; Calculating the attention weight of each group of first aggregate features, and aggregating the multiple groups of first aggregate features based on the attention weight to obtain a second aggregate feature; Key growth parts of the tomato plant are identified based on the second aggregated feature.

5. The method for monitoring growth parameters of greenhouse tomato plants according to claim 1, characterized in that: The step of obtaining target growth parameters of the key growth part based on the three-dimensional reconstruction data corresponding to the key growth part includes: Inputting the three-dimensional reconstruction data corresponding to the key growth part into a target growth parameter recognition model to obtain the target growth parameter output by the target growth parameter recognition model; The key growth parts include the flower of the tomato plant, the leaves of the tomato plant, the stem of the tomato plant and the apex of the tomato plant, and the target growth parameters include the growth period category of the flower, the length of the leaves, the stem thickness of the stem and the height of the apex; The target growth parameter identification model is obtained by training a basic model that matches the target growth parameter identification through the historical three-dimensional reconstruction data of the key growth part and its corresponding target growth parameter label.

6. The method for monitoring growth parameters of greenhouse tomato plants according to claim 1, characterized in that: The obtaining of a future predicted value of the target growth parameter based on the current time series data of the target growth parameter comprises: Based on the current time series data of the target growth parameter, generating the lagged time series data and the moving average time series data of the target growth parameter; splicing the current time series data, the lagged time series data and the moving average time series data to obtain spliced ​​time series data; Convolution and pooling are performed on the spliced ​​time series data to obtain multiple local features of the target growth parameter in the spatiotemporal dimension; Capturing the long-term dependency relationship between each local feature and the local features before and after it, and obtaining a plurality of long-term dependent local features of the target growth parameter in the spatiotemporal dimension; Calculate the attention weights between multiple long-term dependent local features; Based on the attention weights, a hidden state representation of the target growth parameter in the spatiotemporal dimension is calculated with each long-term dependent local feature as the center to obtain multiple hidden state representations; The multiple hidden state representations are mapped to an output space to obtain future predicted values ​​of the target growth parameters.

7. The method for monitoring growth parameters of greenhouse tomato plants according to claim 1, characterized in that: The preprocessing of the initial multimodal data to obtain target multimodal data includes: Performing exposure correction, image enhancement, distortion correction and denoising on the initial RGB image data in sequence to obtain RGB image data to be processed; Performing invalid value filling and denoising on the initial depth image data in sequence to obtain depth image data to be processed; Performing outlier removal, downsampling, and normal vector estimation on the initial three-dimensional point cloud data in sequence to obtain three-dimensional point cloud data to be processed; The RGB image data to be processed, the depth image data to be processed and the three-dimensional point cloud data to be processed are aligned to obtain target RGB image data, target depth image data and target three-dimensional point cloud data.

8. A greenhouse tomato plant growth parameter monitoring device, characterized in that: include: The initial multimodal data acquisition module is used to: acquire the initial multimodal data of tomato plants collected by the collection vehicle in the target greenhouse under multiple viewing angles; The initial multimodal data includes initial RGB image data, initial depth image data and initial three-dimensional point cloud data; A target multimodal data acquisition module is used to: pre-process the initial multimodal data to obtain target multimodal data; A three-dimensional reconstruction data acquisition module is used to perform fusion modeling on the target multimodal data to obtain three-dimensional reconstruction data of the tomato plant; A key growth part identification module, used for identifying the key growth part of the tomato plant based on the three-dimensional reconstruction data; A target growth parameter acquisition module, used to obtain the target growth parameters of the key growth part based on the three-dimensional reconstruction data corresponding to the key growth part; The target growth parameter prediction module is used to obtain the future prediction value of the target growth parameter based on the current time series data of the target growth parameter.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the method for monitoring growth parameters of greenhouse tomato plants according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for monitoring growth parameters of greenhouse tomato plants according to any one of claims 1 to 7 are implemented.

Images