Plant factory plant three-dimensional phenotype information remote acquisition system and method

Abstract

The application discloses a kind of plant factory plant three-dimensional phenotype information remote acquisition system and method.The system includes the plant factory plant acquisition device of photographing plant image;Including the remote terminal of three-dimensional phenotype information after downloading image processing is obtained.It includes: after the multi-view image sequence of plant of plant factory plant acquisition device acquisition is uploaded, remote terminal is downloaded and is input into plant phenotype information acquisition model, and adds grid segmentation label training;After the multi-view image sequence of plant to be measured is uploaded, remote terminal is downloaded and is input into model, and the phenotype information of plant to be measured is output, the remote acquisition of plant factory plant three-dimensional phenotype information is realized.The application device realizes multi-view image remote, fast, automatic acquisition, method can generate high-precision plant three-dimensional model from image sequence faster, realize the acquisition of three-dimensional phenotype information, so that manager can master plant physiological growth condition in real time, can provide technical support for breeding, pest control and other fields.

Description

Technical Field

[0001] This invention relates to a remote information acquisition system, which relates to the fields of smart agriculture and digital agriculture, and specifically to a remote acquisition system and method for three-dimensional phenotypic information of plants in a plant factory. Background Technology

[0002] Plant factories, as a typical representative of the advanced stage of facility agriculture development, aim to meet people's demand for food and plant products by optimizing the plant growth environment, conserving resources, and increasing crop yields. As a highly efficient and stable plant production system, the phenotypic information of plants in plant factories needs to be strictly monitored so that managers can promptly confirm whether environmental conditions have a positive effect on the plants, understand the growth stage of the plants, and make appropriate interventions to ensure the high-efficiency output of the plant factory.

[0003] Currently, traditional methods for acquiring plant phenotypic information mainly rely on manual measurement. This method is not only labor-intensive and costly, but also prone to errors due to variations in human skill and measuring tools. With the development of measurement technology, research has proposed three-dimensional plant phenotypic information acquisition methods based on LiDAR and depth cameras. These methods can acquire plant three-dimensional information non-invasively, but they suffer from high cost and large data volumes. Multi-view stereo vision methods based on monocular cameras are low-cost and simple in structure, but phenotypic information extraction often relies on manually designed thresholds. Furthermore, existing methods for acquiring three-dimensional plant phenotypic information in plant factories often require on-site operation, lacking a remote, rapid, and automated acquisition paradigm. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention provides a remote acquisition system and method for three-dimensional phenotypic information of plants in a plant factory. It establishes a remote transmission link based on a network camera to automatically acquire multi-view image sequences of plants in the plant factory; it rapidly generates high-precision three-dimensional plant models based on neural radiation fields; and it uses deep learning networks for semantic segmentation of plant organs to achieve accurate measurement of three-dimensional phenotypic information of plants within the plant factory. This enables managers to promptly grasp the physiological growth status of plants, ensuring effective management and efficient output of the plant growth process in the plant factory.

[0005] The technical solution adopted in this invention is:

[0006] I. A remote acquisition system for three-dimensional phenotypic information of plants in a plant factory:

[0007] The system includes a plant factory plant acquisition device that captures multi-view image sequences of plants in a plant factory and uploads them to the Internet.

[0008] The system includes a remote terminal for obtaining plant phenotypic information of plant factories by downloading and processing multi-view image sequences of plant factories from the Internet.

[0009] The plant factory plant collection device includes a microcontroller, a network camera, a lifting platform, and an electric turntable. The network camera is installed on the top of the lifting platform, and the plant in the flowerpot is placed on the electric turntable. The network camera is horizontally facing the plant. The microcontroller is wirelessly connected to the network camera, the lifting platform, and the electric turntable. The network camera is connected to a remote terminal network.

[0010] The microcontroller is a NodeMCU-32s module. The core of this module is the ESP32 chip, which integrates functions such as Wi-Fi and Bluetooth. The microcontroller is wirelessly connected to the lifting platform and the electric turntable via Bluetooth, and wirelessly connected to the network camera via Wi-Fi.

[0011] The system incorporates a remote communication mechanism based on On-Screen Display (OSD) technology from network cameras, enabling remote acquisition of plant image sequences in a plant factory. Network cameras capture plant images and automatically upload video to the internet for remote access. Supporting OSD technology, the network cameras can transmit strings alongside the video stream. Terminals can read or modify the network camera's OSD via the Intelligent Security API (ISAPI) protocol, thereby reading or issuing OSD commands.

[0012] The remote terminal is equipped with a plant phenotypic information acquisition model for processing multi-view image sequences of plants in plant factories and acquiring phenotypic information of plants in plant factories.

[0013] II. A method for remotely acquiring phenotypic information in a plant factory's three-dimensional phenotypic information remote acquisition system, comprising:

[0014] 1) Collect multi-view image sequences of several plants using a plant collection device in a plant factory, and then upload them to the Internet. The remote terminal downloads the multi-view image sequences of each plant from the Internet and inputs them into the plant phenotypic information collection model in sequence. After adding grid segmentation labels, the model is trained to obtain a trained plant phenotypic information collection model. The training is completed when the loss function of the plant phenotypic information collection model converges.

[0015] 2) After collecting multi-view image sequences of the plants to be tested using the plant factory plant collection device, the images are uploaded to the Internet. The remote terminal downloads the multi-view image sequences of the plants to be tested from the Internet and inputs them into the trained plant phenotypic information collection model for processing, and then outputs the phenotypic information of the plants to be tested, thereby realizing the remote collection of three-dimensional phenotypic information of plants in the plant factory.

[0016] In practice, a black screen is placed behind the motorized turntable to remove cluttered backgrounds and prevent feature matching failures. The plant pot is used as a reference object; knowing the actual dimensions of the reference object allows us to obtain the actual dimensions of the 3D model.

[0017] In step 1), a multi-view image sequence of several plants is acquired through a plant collection device in a plant factory. Specifically, an OSD command containing information on the lifting height of the lifting platform and the rotation speed of the electric turntable is transmitted to a network camera via a remote terminal. The microcontroller periodically accesses the network camera and receives the OSD command, and then controls the lifting platform to move the network camera to several uniformly spaced specified heights. Each time the camera moves to a specified height, the electric turntable is controlled to rotate the plant one revolution at a specified speed. The network camera captures multi-view images of the plant in real time, thereby obtaining a multi-view image sequence of the plant at various heights.

[0018] In step 1), the plant phenotypic information acquisition model sequentially includes a threshold-based segmentation algorithm, NeRF (Neural Radiation Field), Open3D connected component algorithm, a 3D mesh segmentation network, and the visualization tool library VTK. Each image in the multi-view image sequence of the plant is first segmented by the threshold-based segmentation algorithm to extract the plant background, and then an initial 3D mesh model of the plant is generated based on NeRF. The initial 3D mesh model is then preprocessed based on the connected component algorithm to obtain a preprocessed 3D mesh model of the plant. Mesh segmentation labels are then added to the preprocessed 3D mesh model of the plant, and then it is input into the 3D mesh segmentation network for processing. The visualization tool library VTK finally realizes the measurement of phenotypic information at the organ level of the plant, including: plant height and width, leaf tilt angle and average leaf tilt angle, leaf area of ​​a single leaf and all leaves, and volume of leaves, stems, flowers, and fruits.

[0019] Based on NeRF, coarse geometric and color texture features are generated, and a rough surface mesh is extracted. This process involves refining the color textures of vertices, triangles, and surfaces. NeRF updates the mesh until the loss function converges, obtaining an initial 3D mesh model of the plant. The model is preprocessed using the Open3D connected component algorithm to remove outliers generated during reconstruction, thus eliminating noise at the exterior or edges of the generated initial plant mesh model due to the influence of camera imaging quality and background environment.

[0020] The 3D mesh segmentation network includes a feature sampling layer, three convolutional pooling modules, and four convolutional upsampling modules. The inputs of the 3D mesh segmentation network are sequentially fed into the feature sampling layer, the three convolutional pooling modules, and the first convolutional upsampling module for processing. The outputs of the second convolutional pooling module and the first convolutional upsampling module are jointly fed into the second convolutional upsampling module for processing. The outputs of the first convolutional pooling module and the second convolutional upsampling module are jointly fed into the third convolutional upsampling module for processing, and then fed into the fourth convolutional upsampling module for further processing. The processed outputs serve as the segmentation results of the 3D mesh segmentation network.

[0021] The 3D mesh segmentation network approximates the U-Net architecture, performing feature extraction and semantic segmentation through three layers of downsampling and upsampling. In the initial stage, feature vectors from all faces of the mesh model are extracted and input into the network via a feature sampling layer. In a single convolutional module, the input passes through a convolutional layer, a convolutional layer, and an attention layer sequentially; these three layers are connected by residual connections. Residual connections simplify the learning process, enhance gradient propagation, and improve model performance and generalization ability. The ECA (efficient channel attention) module avoids the side effects of dimensionality reduction on channel attention and maintains model performance through appropriate local cross-channel interactions. Upsampling is the inverse process of pooling; it restores the mesh structure by increasing feature resolution, employing a bilinear upsampling method to provide smoother interpolation than the most recent upsampling method.

[0022] The feature sampling layer specifically extracts feature vectors for each triangular mesh region from the preprocessed 3D mesh model of the plant using a mesh feature extraction method. This includes the relationship features between the triangular mesh region and its adjacent triangular mesh regions, as well as the surface features of the triangular mesh region. Specifically, the relationship features between the triangular mesh region and its three adjacent triangular mesh regions are sorted from largest to smallest by the length of their adjacent edges. The sorted three adjacent triangular mesh regions are then used as the relationship features, ensuring invariance to rotation, translation, and scale. The three adjacent triangular mesh regions are designated as the first, second, and third adjacent regions according to the sorting order. The surface features of the triangular mesh region include six dihedral angles, ... The system contains one area, three interior angles, and color texture information. The six dihedral angles include the dihedral angles between the triangular mesh region and three adjacent triangular mesh regions, as well as the dihedral angles between the three adjacent triangular mesh regions themselves. These six dihedral angles are ordered sequentially, with the dihedral angles between the triangular mesh region and the first, second, and third adjacent regions respectively designated as the first, second, and third dihedral angles. The dihedral angles between the first, second, and third adjacent regions themselves are designated as the fourth, fifth, and sixth dihedral angles. The area is the normalized area of ​​the triangular mesh region. The three interior angles are the three interior angles of the triangular mesh region, ordered from largest to smallest. The color texture information includes the red channel (R), green channel (G), and blue channel (B) information.

[0023] The normalized area of ​​the triangular mesh region is as follows:

[0024]

[0025] Where A' is the normalized area of ​​the triangular mesh region, n is the number of triangular mesh regions in the preprocessed 3D mesh model of the plant, and A is the actual area of ​​the triangular mesh region; A i This represents the actual area of ​​the i-th triangular mesh region in the preprocessed 3D mesh model of the plant.

[0026] The convolutional fusion pooling module includes a first convolutional layer, a second convolutional layer, an efficient channel attention layer (ECA), and an average pooling layer. The inputs to the convolutional fusion pooling module are sequentially input into the first convolutional layer and the second convolutional layer for processing. The outputs of the first convolutional layer and the second convolutional layer are jointly input into the efficient channel attention layer (ECA) for processing and then input into the average pooling layer. The output of the average pooling layer is used as the output of the convolutional fusion pooling module.

[0027] The convolutional fusion upsampling module includes a third convolutional layer, a fourth convolutional layer, an efficient channel attention layer (ECA), and a bilinear upsampling layer. The inputs of the convolutional fusion upsampling module are sequentially input into the third and fourth convolutional layers for processing. The outputs of the third and fourth convolutional layers are jointly input into the efficient channel attention layer (ECA) for further processing before being input into the bilinear upsampling layer. The output of the bilinear upsampling layer serves as the output of the convolutional fusion upsampling module.

[0028] The number of facets is changed by setting pooling or upsampling layers after each convolutional layer. Deep and shallow modules of the same scale are concatenated together via residual connections to fuse the positional information of the shallow layer with the semantic information of the deep layer. In the overall segmentation network, the number of channels of the convolutional modules is set to 64, 128, 256, 256, 128, and 64, respectively. The pooling layer sizes are 1200, 900, and 300, respectively, and the upsampling layer sizes are 300, 900, and 1200, respectively.

[0029] The average pooling layer specifically involves first using the L2 norm to obtain the feature intensity of each triangular mesh region in the preprocessed 3D mesh model of the plant, based on each triangular mesh region and its adjacent triangular mesh regions, as detailed below:

[0030]

[0031] Among them, w i The feature intensity of the i-th triangular mesh region in the preprocessed 3D mesh model of the plant.

[0032] v i 0 v represents the feature vector of the i-th triangular mesh region in the 3D mesh model of plant preprocessing. i m This represents the feature vector of the m-th adjacent triangular mesh region in the i-th triangular mesh region of the plant preprocessing 3D mesh model.

[0033] After obtaining the feature intensities of each triangular mesh region, they are sorted in ascending order. Each triangular mesh region is then pooled in ascending order of its feature intensities. Specifically, pooling involves taking the average of the feature vectors of each triangular mesh region and its three adjacent triangular mesh regions as the feature intensity of the current triangular mesh region. Finally, the pooling result is output after processing by an average pooling layer. The pooling result retains the features of the original surface as much as possible. After pooling, the relationship between the surface and its adjacent surfaces needs to be recalculated, and non-manifold surfaces are not allowed to be formed by pooling.

[0034] The pooling operation of the grid includes finding pooling regions, merging pooling region features, and redefining the adjacency of merged features.

[0035] The beneficial effects of this invention are:

[0036] This invention presents a remote plant phenotypic acquisition method based on the OSD (OnScreenDisplay) mechanism of a network camera. It remotely controls a plant phenotypic acquisition device deployed inside a plant factory, acquiring multi-view image sequences of plants at different heights and angles by adjusting the height of the lifting platform and the angle of the electric turntable. This method achieves remote, rapid, and automated acquisition of multi-view images, saving labor costs and facilitating subsequent extraction of three-dimensional phenotypic information.

[0037] This invention proposes a fast and high-precision method for generating 3D plant mesh models. A two-stage method based on Neural Radiance Fields (NeRF) is designed to generate 3D plant mesh models. After generating coarse geometric and color texture features based on NeRF, an iterative mesh refinement algorithm is used to optimize the vertices, faces, and color textures of the 3D mesh model. Compared to traditional methods, NeRF, through its implicit representation of the 3D scene, can generate high-precision 3D plant models more quickly from 2D multi-view image sequences.

[0038] This invention proposes a semantic segmentation method for organ-level structures in a 3D mesh model of plants. A method for extracting surface feature vectors from the 3D mesh is designed, along with a 3D mesh segmentation network incorporating an attention mechanism. Semantic prediction for each facet of the 3D mesh is achieved through an approximate U-Net architecture, resulting in good semantic segmentation performance. Based on the segmentation results provided by the model, a method for measuring plant phenotypic information is designed, enabling managers to monitor the physiological growth status of plants in real time. Furthermore, by recording the temporal changes in plant 3D phenotypic information, technical support can be provided for breeding, pest and disease control, and other fields. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the plant phenotypic acquisition device of the present invention;

[0040] Figure 2 This is a schematic diagram of the mesh surface and its features in the three-dimensional mesh model of the present invention;

[0041] Figure 3 This is a schematic diagram of the pooling and upsampling operations of the plant 3D mesh model segmentation network of the present invention.

[0042] Figure 4 This is a schematic diagram of the segmentation network framework for the three-dimensional mesh model of the plant according to the present invention;

[0043] Figure 5 This is a schematic diagram of the overall process of the remote monitoring system for plant growth in a plant factory according to the present invention.

[0044] In the picture: 1. Microcontroller, 2. Network camera, 3. Lifting platform, 4. Plant, 5. Electric turntable. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] The plant factory plant three-dimensional phenotypic information remote acquisition system of the present invention includes a plant factory plant acquisition device for capturing multi-view image sequences of plant factories and uploading them to the Internet; and a plant factory plant phenotypic acquisition remote terminal for downloading and processing multi-view image sequences of plant factories from the Internet to obtain three-dimensional phenotypic information of plant factories.

[0047] like Figure 1 As shown, the plant collection device in the plant factory includes a microcontroller 1, a network camera 2, a lifting platform 3, and an electric turntable 5. The network camera 2 is installed on the top of the lifting platform 3. Plants 4 planted in flowerpots are placed on the electric turntable 5. The network camera 2 is horizontally facing the plant 4. The microcontroller 1 is wirelessly connected to the network camera 2, the lifting platform 3, and the electric turntable 5 respectively. The network camera 2 is connected to a remote terminal network.

[0048] The microcontroller is a NodeMCU-32s module. The core of this module is the ESP32 chip, which integrates Wi-Fi, Bluetooth and other functions. Microcontroller 1 is wirelessly connected to the lifting platform 3 and the electric turntable 5 via Bluetooth, and microcontroller 1 is wirelessly connected to the network camera 2 via Wi-Fi.

[0049] The system incorporates a remote communication mechanism based on On-Screen Display (OSD) technology from network cameras, enabling remote acquisition of plant image sequences in a plant factory. Network camera 2 is used to capture plant images and automatically uploads the video to the internet for remote access. Simultaneously, network camera 2 supports OSD technology, allowing the transmission of strings alongside the video stream. Terminals can read or modify the OSD of network camera 2 via the Intelligent Security API (ISAPI) protocol, thereby reading or issuing OSD commands.

[0050] The remote terminal is equipped with a plant phenotypic information acquisition model for processing multi-view image sequences of plants in plant factories and acquiring phenotypic information of plants in plant factories.

[0051] like Figure 5 As shown, the remote acquisition method of phenotypic information of the plant factory plant three-dimensional phenotypic information remote acquisition system of the present invention includes:

[0052] 1) Collect a series of multi-view images of several plants 4 using a plant collection device in a plant factory, and then upload them to the Internet. The remote terminal downloads the multi-view image sequences of each plant 4 from the Internet and inputs them into the plant phenotypic information collection model in sequence. After adding grid segmentation labels, the model is trained to obtain a trained plant phenotypic information collection model. The training is completed when the loss function of the plant phenotypic information collection model converges.

[0053] In practice, a black screen is placed behind the motorized turntable 5 to remove cluttered backgrounds and avoid feature matching failures. The plant pot is used as a reference object; given the actual dimensions of the reference object, the actual dimensions of the 3D model can be obtained.

[0054] In step 1), a series of multi-view images of several plants 4 are collected by the plant collection device in the plant factory. Specifically, OSD commands containing the lifting height of the lifting platform 3 and the rotation speed of the electric turntable 5 are transmitted to the network camera 2 through a remote terminal. The microcontroller 1 periodically accesses the network camera 2 and receives the OSD commands. Then, it controls the lifting platform 3 to move the network camera 2 to several uniformly spaced specified heights. Each time the camera moves to a specified height, the electric turntable 5 is controlled to rotate the plant 4 one revolution at a specified speed. The network camera 2 captures multi-view images of the plant 4 in real time, thereby obtaining a series of multi-view images of the plant 4 at various heights.

[0055] In step 1), the plant phenotypic information acquisition model sequentially includes a threshold-based segmentation algorithm, NeRF (Neural Radiation Field), Open3D connected component algorithm, a 3D mesh segmentation network, and the visualization tool library VTK. Each image in the multi-view image sequence of plant 4 is first segmented by the threshold-based segmentation algorithm to extract the background of plant 4. Then, the initial 3D mesh model of plant 4 is generated based on NeRF. The initial 3D mesh model is then preprocessed based on the connected component algorithm to obtain a preprocessed 3D mesh model of plant 4. Mesh segmentation labels are then added to the preprocessed 3D mesh model of plant 4, and then it is input into the 3D mesh segmentation network for processing. Finally, the visualization tool library VTK measures the phenotypic information of plant 4 at the organ level, including: the height and width of plant 4, the leaf tilt angle and average leaf tilt angle, the leaf area of ​​a single leaf and all leaves, and the volume of leaves, stems, flowers, and fruits.

[0056] Based on NeRF, coarse geometric and color texture features are generated, and a rough surface mesh is extracted. This process involves refining the color textures of vertices, triangles, and surfaces. NeRF updates the mesh until the loss function converges, obtaining an initial 3D mesh model of the plant. The model is preprocessed using the Open3D connected component algorithm to remove outliers generated during reconstruction, thus eliminating noise at the exterior or edges of the generated initial plant mesh model due to the influence of camera imaging quality and background environment.

[0057] The 3D mesh segmentation network includes a feature sampling layer, three convolutional pooling modules, and four convolutional upsampling modules. The inputs of the 3D mesh segmentation network are sequentially fed into the feature sampling layer, the three convolutional pooling modules, and the first convolutional upsampling module for processing. The outputs of the second convolutional pooling module and the first convolutional upsampling module are jointly fed into the second convolutional upsampling module for processing. The outputs of the first convolutional pooling module and the second convolutional upsampling module are jointly fed into the third convolutional upsampling module for processing, and then fed into the fourth convolutional upsampling module for further processing. The processed outputs serve as the segmentation result of the 3D mesh segmentation network.

[0058] The 3D mesh segmentation network approximates the U-Net architecture, performing feature extraction and semantic segmentation through three layers of downsampling and upsampling. In the initial stage, feature vectors from all faces of the mesh model are extracted and input into the network via a feature sampling layer. In a single convolutional module, the input passes through a convolutional layer, a convolutional layer, and an attention layer sequentially; these three layers are connected by residual connections. Residual connections simplify the learning process, enhance gradient propagation, and improve model performance and generalization ability. The ECA (efficient channel attention) module avoids the side effects of dimensionality reduction on channel attention and maintains model performance through appropriate local cross-channel interactions. Upsampling is the inverse process of pooling; it restores the mesh structure by increasing feature resolution, employing a bilinear upsampling method to provide smoother interpolation than the most recent upsampling method.

[0059] The feature sampling layer specifically uses a mesh feature extraction method to extract feature vectors for each triangular mesh region in the preprocessed 3D mesh model of the plant. This includes the relationship features between the triangular mesh region and its adjacent triangular mesh regions, as well as the surface features of the triangular mesh region. The relationship features between the triangular mesh region and its three adjacent triangular mesh regions are specifically sorted by the length of their adjacent edges from largest to smallest. The three adjacent triangular mesh regions are then used as the relationship features, ensuring rotation, translation, and scale invariance. The three adjacent triangular mesh regions are designated as the first, second, and third adjacent regions according to the sorting order. The surface features of the triangular mesh region include six dihedral angles and one... The data includes area, three interior angles, and color texture information. The six dihedral angles are the dihedral angles between a triangular mesh region and three adjacent triangular mesh regions, as well as the dihedral angles between three adjacent triangular mesh regions. These six dihedral angles are ordered sequentially, with the dihedral angles between the triangular mesh region and the first, second, and third adjacent regions respectively designated as the first, second, and third dihedral angles. The dihedral angles between the first, second, and third adjacent regions are designated as the fourth, fifth, and sixth dihedral angles. The area is the normalized area of ​​the triangular mesh region. The three interior angles are the three interior angles of the triangular mesh region, ordered from largest to smallest. The color texture information includes the red channel (R), green channel (G), and blue channel (B) information.

[0060] The normalized area of ​​the triangular mesh region is as follows:

[0061]

[0062] Where A' is the normalized area of ​​the triangular mesh region, n is the number of triangular mesh regions in the preprocessed 3D mesh model of the plant, and A is the actual area of ​​the triangular mesh region; A i This represents the actual area of ​​the i-th triangular mesh region in the preprocessed 3D mesh model of the plant.

[0063] The convolutional fusion pooling module includes a first convolutional layer, a second convolutional layer, an efficient channel attention layer (ECA), and an average pooling layer. The inputs to the convolutional fusion pooling module are sequentially fed into the first convolutional layer and the second convolutional layer for processing. The outputs from the first convolutional layer and the second convolutional layer are fed into the efficient channel attention layer (ECA) for further processing before being fed into the average pooling layer. The output of the average pooling layer serves as the output of the convolutional fusion pooling module.

[0064] The convolutional fusion upsampling module includes a third convolutional layer, a fourth convolutional layer, an efficient channel attention layer (ECA), and a bilinear upsampling layer. The inputs of the convolutional fusion upsampling module are sequentially fed into the third and fourth convolutional layers for processing. The outputs of the third and fourth convolutional layers are fed into the efficient channel attention layer (ECA) for further processing before being fed into the bilinear upsampling layer. The output of the bilinear upsampling layer serves as the output of the convolutional fusion upsampling module.

[0065] The number of facets is changed by setting pooling or upsampling layers after each convolutional layer. Deep and shallow modules of the same scale are concatenated together via residual connections to fuse the positional information of the shallow layer with the semantic information of the deep layer. In the overall segmentation network, the number of channels of the convolutional modules is set to 64, 128, 256, 256, 128, and 64, respectively. The pooling layer sizes are 1200, 900, and 300, respectively, and the upsampling layer sizes are 300, 900, and 1200, respectively.

[0066] The average pooling layer specifically involves first using the L2 norm to obtain the feature intensity of each triangular mesh region in the preprocessed 3D mesh model of the plant, based on each triangular mesh region and its adjacent triangular mesh regions, as detailed below:

[0067]

[0068] Among them, w i The feature intensity of the i-th triangular mesh region in the preprocessed 3D mesh model of the plant.

[0069] v i 0 v represents the feature vector of the i-th triangular mesh region in the 3D mesh model of plant preprocessing. i m This represents the feature vector of the m-th adjacent triangular mesh region in the i-th triangular mesh region of the plant preprocessing 3D mesh model.

[0070] After obtaining the feature intensities of each triangular mesh region, they are sorted in ascending order. Each triangular mesh region is then pooled in ascending order of its feature intensities. Specifically, pooling involves taking the average of the feature vectors of each triangular mesh region and its three adjacent triangular mesh regions as the feature intensity of the current triangular mesh region. Finally, the pooling result is output after processing by an average pooling layer. The pooling result retains the features of the original surface as much as possible. After pooling, the relationship between the surface and its adjacent surfaces needs to be recalculated, and non-manifold surfaces are not allowed to be formed by pooling.

[0071] The pooling operation for a grid includes finding pooling regions, merging pooling region features, and redefining the adjacency of merged features, such as... Figure 3 As shown.

[0072] 2) After collecting multi-view image sequences of the plant under test 4 through the plant factory plant collection device, the images are uploaded to the Internet. The remote terminal downloads the multi-view image sequences of the plant under test 4 from the Internet and inputs them into the trained plant phenotypic information collection model for processing and outputting the phenotypic information of the plant under test 4, thereby realizing the remote collection of three-dimensional phenotypic information of the plant in the plant factory.

[0073] Specific embodiments of the present invention are as follows:

[0074] To achieve a balance between image acquisition quality and speed, this embodiment sets the rotation speed of the electric turntable 5 of the plant acquisition device in the plant factory to 30 seconds per revolution, and the height of the lifting platform 3 is divided into three levels: 10cm, 20cm, and 30cm. This embodiment uses the plant's flowerpot as a reference, with a pot diameter of 50cm.

[0075] Upon receiving the acquisition signal from the remote terminal, microcontroller 1 first sends a command to the lifting platform 3 via Bluetooth, instructing it to rise to a height of 30cm, allowing network camera 2 to capture the upper part of plant 4. Subsequently, microcontroller 1 sends a command to the electric turntable 5, causing it to rotate at a constant speed of 30 seconds per revolution. After the electric turntable 5 completes one revolution, microcontroller 1 commands the lifting platform 3 to descend to a height of 20cm, allowing network camera 2 to capture the middle part of plant 4. After repeating the turntable rotation operation, microcontroller 1 again commands the lifting platform 3 to descend to a height of 10cm, repeating the turntable operation. During the triggered actions, the remote terminal retrieves the video stream from network camera 2 in real time and extracts keyframes at certain time intervals, ensuring that 40 images are captured for each revolution of electric turntable 5, resulting in a total of 120 images throughout the acquisition process. This yields a multi-view image sequence of the plant at different heights and angles.

[0076] Subsequently, the image sequences were processed using a plant phenotypic information acquisition model. In the first stage, coarse geometric and color texture features were generated based on NeRF. In the second stage, the coarse surface mesh extracted from the NeRF model in the first stage was optimized using an iterative mesh refinement algorithm. This process involved refining the color textures of vertices, triangles, and surfaces. The first stage involved training for 30,000 steps, evaluating approximately 218 points per step, with a learning rate that decayed exponentially from 0.01 to 0.001. In the second stage, training was conducted for 10,000 to 30,000 steps based on the convergence of the first stage, with a learning rate set to 0.0001. Both stages used the Adam (adaptive moment estimation) optimizer.

[0077] Then, based on Open3D's connected component algorithm (cluster_connected_triangles), the number of triangle faces in each cluster of the generated mesh model is calculated, and triangles with fewer than 100 faces are discarded. This preprocessing algorithm can remove outliers generated during the reconstruction process.

[0078] Subsequently, semantic segmentation at the organ level was performed on the plant based on the 3D mesh segmentation model, i.e., mesh segmentation labels were added. This semantic segmentation algorithm is a supervised algorithm, therefore a dataset needs to be built and the samples manually labeled. 3D mesh model datasets of different varieties and growth stages of plants were pre-collected, and the leaves, stems, flowers, fruits, flowerpots, and soil of the plant models were labeled using Blender software. The dataset was divided into training and testing sets in a 7:3 ratio. The semantic segmentation network was iteratively trained using the training set, and the semantic segmentation performance of the model was tested using the testing set.

[0079] In this embodiment, the generated mesh is a triangular mesh. To input the mesh model into the semantic segmentation network, a mesh feature extraction method needs to be designed. For example... Figure 2 As shown, a single face (face O) of the mesh has at most three circumscribed faces (face A, face B, and face C). The normal vectors of faces O, A, B, and C are respectively... To ensure invariance to rotation, translation, and scale, the relationship between faces and adjacent faces needs to be represented in an ordered manner. Therefore, this embodiment sorts adjacent faces by the length of their adjacent edges. For example... Figure 2 In the middle, the order should be C side, A side, B side.

[0080] In this embodiment, each face of the mesh features six dihedral angles, one area, three interior angles, and three color channels. The six dihedral angles follow the sorting method described above and include: the angle θ between face O and face C. OC The angle θ between plane O and plane A OA The angle θ between plane O and plane B OB The angle θ between surface A and surface B AB The angle θ between surface A and surface C AC The angle θ between surface B and surface C BC An area refers to the normalized area of ​​face O across all faces in the mesh model. According to the sorting method described above, the three interior angles should be sorted from smallest to largest, including: the interior angle α between face A and face B. AB The interior angle α between face B and face C BC The interior angle α between face A and face C AC Furthermore, the surface features also include color texture information, including the red channel R, green channel G, and blue channel B. Therefore, the surface feature vector v is a thirteen-dimensional vector (θ... OC ,θ OA ,θOB ,θ AB ,θ AC ,θ BC ,A',α AB ,α BC ,α AC ,R,G,B).

[0081] Upsampling uses bilinear interpolation. The feature vector of the newly generated surface is formed by summing the feature vectors of the original adjacent surfaces (weight 0.25) and the feature vectors of the original surfaces (weight 0.5). Figure 3 As shown.

[0082] like Figure 4 The diagram shows the structure of the 3D mesh segmentation network. In this embodiment, the initial learning rate is set to 0.001, decreasing by a factor of 10 after every 50 iterations. The loss function is set to the cross-entropy function. The Adam method is selected as the training optimizer, with the batch size and maximum training iterations set to 16 and 200, respectively. After training and testing, the model with the best performance is selected as the plant 3D mesh segmentation model. The plant 3D mesh model obtained through remote transmission in the previous steps is input into the plant 3D mesh segmentation model to obtain the semantic prediction results for each face of the 3D mesh model.

[0083] This implementation uses VTK to measure phenotypic information at the plant organ level. The `GetBounds()` function in `vtkPolyData` retrieves the parameters of the 3D mesh bounding box, specifically the maximum and minimum values ​​along the three coordinate axes. `vtkPolyData` is used to obtain the smallest cube enclosing the 3D mesh of the plant (excluding the pot and soil), thus determining the plant height and width. `vtkPolyData` is also used to obtain the leaf ventral normal, calculate its angle with the stem direction, and obtain the leaf inclination angle; averaging across all leaves yields the average leaf inclination angle. `vtkMassProperties` can calculate the surface area and volume of closed triangular meshes. `vtkMassProperties` calculates the leaf area of ​​a single leaf and all leaves; it also calculates the volume of leaves, stems, flowers, and fruits, and correlates this with the plant's biomass.

[0084] All of the above plant phenotypic information is stored in the database. Researchers or managers can use plant phenotypic information from the same time period to understand the current physiological growth status of the plant; they can also use plant phenotypic information from multiple time periods to study the temporal changes of the plant throughout its growth process, thereby providing technical support for breeding, pest and disease control, and other fields.

Claims

1. A method for remotely acquiring phenotypic information in a plant factory's three-dimensional phenotypic information remote acquisition system, characterized in that, include: 1) Collect a series of multi-view images of several plants (4) using a plant factory plant collection device, and then upload them to the Internet. The remote terminal downloads the multi-view image sequences of each plant (4) from the Internet and inputs them into the plant phenotypic information collection model in sequence. After adding grid segmentation labels, the model is trained to obtain the plant phenotypic information collection model that has been trained. 2) After collecting the multi-view image sequence of the plant to be tested (4) through the plant factory plant collection device, the image sequence is uploaded to the Internet. The remote terminal downloads the multi-view image sequence of the plant to be tested (4) from the Internet and inputs it into the trained plant phenotypic information collection model for processing and outputs the phenotypic information of the plant to be tested (4), thereby realizing the remote collection of three-dimensional phenotypic information of the plant in the plant factory. In step 1), the plant phenotypic information acquisition model includes, in sequence, a threshold-based segmentation algorithm, NeRF (Neural Radiation Field), a connected component algorithm, a three-dimensional mesh segmentation network, and a visualization tool library VTK; the three-dimensional mesh segmentation network includes a feature sampling layer, three convolutional fusion pooling modules, and four convolutional fusion upsampling modules. The feature sampling layer specifically extracts feature vectors for each triangular mesh region from the pre-processed 3D mesh model of the plant using a mesh feature extraction method. These vectors include the relationship features between the triangular mesh region and its adjacent triangular mesh regions, as well as the surface features of the triangular mesh region. Specifically, the relationship features between the triangular mesh region and its three adjacent triangular mesh regions are sorted from largest to smallest by the length of their adjacent sides. The three adjacent triangular mesh regions are then designated as the first, second, and third adjacent regions according to their sorting order. The surface features of the triangular mesh region include six dihedral angles. The system contains: an area, three interior angles, and color texture information. The six dihedral angles include those between a triangular mesh region and three adjacent triangular mesh regions, as well as those between three adjacent triangular mesh regions themselves. These six dihedral angles are ordered sequentially, with the dihedral angles between the triangular mesh region and its first, second, and third adjacent regions respectively designated as the first, second, and third dihedral angles. The dihedral angles between the first, second, and third adjacent regions themselves are designated as the fourth, fifth, and sixth dihedral angles. The area is the normalized area of ​​the triangular mesh region. The three interior angles are the three interior angles of the triangular mesh region, ordered from largest to smallest. The color texture information includes the red channel. Green Channel and the blue channel information.

2. The method for remotely acquiring phenotypic information of a plant factory plant three-dimensional phenotypic information remote acquisition system according to claim 1, characterized in that: In step 1), a multi-view image sequence of several plants (4) is collected by the plant collection device in the plant factory. Specifically, the OSD command containing the lifting height of the lifting platform (3) and the rotation speed of the electric turntable (5) is transmitted to the network camera (2) through the remote terminal. The microcontroller (1) accesses the network camera (2) and receives the OSD command. Then, the lifting platform (3) is controlled to move the network camera (2) to several uniformly spaced specified heights. Each time the camera moves to a specified height, the electric turntable (5) is controlled to drive the plant (4) to rotate one revolution at a specified speed. The network camera (2) captures multi-view images of the plant (4) in real time, thereby obtaining a multi-view image sequence of the plant (4) at various heights.

3. The method for remotely acquiring phenotypic information of a plant factory plant three-dimensional phenotypic information remote acquisition system according to claim 1, characterized in that: Each image in the multi-view image sequence of the plant (4) is first segmented by a threshold-based segmentation algorithm to extract the background of the plant (4), and then an initial three-dimensional mesh model of the plant (4) is generated based on the neural radiation field NeRF. The initial three-dimensional mesh model of the plant is then preprocessed based on the connected component algorithm to obtain a preprocessed three-dimensional mesh model of the plant. Then, mesh segmentation labels are added to the preprocessed three-dimensional mesh model of the plant, and then it is input into the three-dimensional mesh segmentation network for processing. The visualization tool library VTK ultimately enables the measurement of phenotypic information at the organ level of the plant (4), including: the height and width of the plant (4), the leaf tilt angle and average leaf tilt angle of the leaves, the leaf area of ​​a single leaf and all leaves, and the volume of leaves, stems, flowers and fruits.

4. The method for remotely acquiring phenotypic information of a plant factory plant three-dimensional phenotypic information remote acquisition system according to claim 3, characterized in that: The input of the 3D mesh segmentation network is sequentially fed into the feature sampling layer, three convolutional fusion pooling modules, and the first convolutional fusion upsampling module for processing. The outputs of the second convolutional fusion pooling module and the first convolutional fusion upsampling module are jointly fed into the second convolutional fusion upsampling module for processing. The outputs of the first convolutional fusion pooling module and the second convolutional fusion upsampling module are jointly fed into the third convolutional fusion upsampling module for processing, and then fed into the fourth convolutional fusion upsampling module for further processing. The processed output is used as the segmentation result of the 3D mesh segmentation network.

5. The method for remotely acquiring phenotypic information of a plant factory plant three-dimensional phenotypic information remote acquisition system according to claim 1, characterized in that: The normalized area of ​​the triangular mesh region is as follows: in, A´ The normalized area of ​​the triangular grid region. n The number of triangular mesh regions in the preprocessed 3D mesh model of the plant; A This represents the actual area of ​​the triangular grid region; A i The first in the three-dimensional mesh model for preprocessing plants i The actual area of ​​each triangular grid region.

6. The method for remotely acquiring phenotypic information of a plant factory plant three-dimensional phenotypic information remote acquisition system according to claim 4, characterized in that: The convolutional fusion pooling module includes a first convolutional layer, a second convolutional layer, an efficient channel attention layer (ECA), and an average pooling layer. The inputs of the convolutional fusion pooling module are sequentially input into the first convolutional layer and the second convolutional layer for processing. The outputs of the first convolutional layer and the second convolutional layer are jointly input into the efficient channel attention layer (ECA) for processing and then input into the average pooling layer. The output of the average pooling layer is used as the output of the convolutional fusion pooling module. The convolutional fusion upsampling module includes a third convolutional layer, a fourth convolutional layer, an efficient channel attention layer (ECA), and a bilinear upsampling layer. The inputs of the convolutional fusion upsampling module are sequentially input into the third and fourth convolutional layers for processing. The outputs of the third and fourth convolutional layers are jointly input into the efficient channel attention layer (ECA) for further processing before being input into the bilinear upsampling layer. The output of the bilinear upsampling layer serves as the output of the convolutional fusion upsampling module.

7. The method for remotely acquiring phenotypic information of a plant factory plant three-dimensional phenotypic information remote acquisition system according to claim 6, characterized in that: The average pooling layer specifically involves first using the L2 norm to obtain the feature intensity of each triangular mesh region in the preprocessed 3D mesh model of the plant, based on each triangular mesh region and its adjacent triangular mesh regions, as detailed below: in, w i The first in the three-dimensional mesh model for preprocessing plants i The characteristic intensity of each triangular mesh region; This represents the first element in the 3D mesh model of plant pretreatment. i Feature vectors of triangular mesh regions This represents the first element in the 3D mesh model of plant pretreatment. i The first triangular grid region m Feature vectors of adjacent triangular mesh regions; After obtaining the feature intensities of each triangular mesh region, they are sorted in ascending order. Each triangular mesh region is then pooled in ascending order of its feature intensities. Specifically, pooling involves taking the average of the feature vectors of each triangular mesh region and its three adjacent triangular mesh regions as the feature intensity of the current triangular mesh region. Finally, the pooling result is output after processing by an average pooling layer.

Images