Method and device for constructing digital twinborn map of track type high-flux crop phenotype platform

Abstract

The invention provides a track type high-flux crop phenotype platform digital twinborn map construction method and device, and the method comprises the steps: collecting multi-source phenotype data of a planting land parcel region according to a constructed track phenotype platform, and constructing a track phenotype platform twinborn body in a digital twinborn space; preprocessing the multi-source phenotype data to obtain three-dimensional point cloud data and plant phenotype preprocessing data of the planting plot area; according to the plant phenotype preprocessing data, constructing to obtain a plant twinborn body; and constructing a regional environment twinborn body of the planting plot region, and integrating the track phenotype platform twinborn body, the plant twinborn body and the regional environment twinborn body in the digital twinborn space to obtain a digital twinborn map. By means of the plant phenotype digital twinning construction method and device, the defects that during plant phenotype digital twinning construction, digital twinning construction of a plant body is lacked, and fusion application of multi-modal data collected on a phenotype platform in a unified twinning space is lacked are overcome.

Description

Technical Field

[0001] The present invention relates to the field of plant phenotyping technology, and in particular to a method and device for constructing a digital twin map of a track-type high-throughput crop phenotyping platform. Background Art

[0002] Digital twin is a technical means that integrates multi-physics, multi-scale, and multi-disciplinary attributes, has real-time synchronization, faithful mapping, and high fidelity, and can realize the interaction and integration of the physical world and the information world. In the field of agriculture, the application objects of digital twin technology are currently mainly focused on equipment and environmental entities, accurately mapping the monitored plants in the digital world, building a digital map of plant planting, and realizing the implementation of the concept of plant digital twin. Plants are planted and managed in the virtual space, and a digital twin map of crops is built. The most important thing is to map the plant's time, space, morphology, attributes and other information in the virtual twin space. That is, when constructing a virtual space for production scenes, the position and morphology information of plants in the real world are restored in the virtual space, and the real-time data of the plant environment and the machine are provided with the support of the plant factory to drive the digital twin base to run, measure, and predict in real time, so as to form a parallel plant management mechanism synchronized with the real world.

[0003] The application of digital twin technology in the field of agricultural plant phenotyping is currently in its infancy. As for the construction of digital twins of orbital high-throughput crop phenotyping platforms, existing technical means mainly focus on the equipment level, lacking the construction of digital twins of the plant itself. The digital twin of the plant itself is a superposition of time, space, and phenotypic information, and also puts forward high requirements for planting accuracy (group-plot-cell-single plant). Secondly, the multi-source phenotypic data collected by the phenotyping platform are multi-modal cross-domain big data, lacking the analysis and fusion application of multi-modal data in a unified twin space. Summary of the invention

[0004] The present invention provides a method and device for constructing a digital twin map of a track-type high-throughput crop phenotyping platform, so as to overcome the defects that in the process of constructing the digital twin of plant phenotypes, there is a lack of a digital twin construction process for the plant body, and the multimodal data collected on the phenotyping platform lacks the defects of being integrated and applied in a unified twin space.

[0005] The present invention provides a method for constructing a digital twin map of a track-type high-throughput crop phenotyping platform, the method comprising the following steps: Collect multi-source phenotypic data of the planting area based on the constructed track phenotyping platform, and build a track phenotyping platform twin in the digital twin space; Preprocessing the multi-source phenotypic data to obtain three-dimensional point cloud data of the planting plot area and plant phenotypic preprocessing data; Determine the spatial coordinates of individual plants in the planting plot area in the digital twin space according to the plant phenotypic preprocessing data, perform three-dimensional reconstruction on the three-dimensional point cloud data to obtain a cell map model of the planting plot area, and map the spatial coordinates and the cell map model to the digital twin space to obtain a twin space model; Performing multi-source data fusion on the plant phenotype preprocessing data to obtain multi-source fusion data, and constructing a plant grid model based on the multi-source fusion data; Analyzing the multi-source fusion data to obtain phenotypic indicators, and superimposing the phenotypic indicators and the plant grid model on the twin space model to obtain plant twins; A regional environmental twin of the planting plot area is constructed, and the track phenotype platform twin, the plant twin, and the regional environment twin are integrated in the digital twin space to obtain a digital twin map.

[0006] In some embodiments, constructing a track phenotype platform twin in a digital twin space includes: According to the physical structure of the track phenotype platform, a three-dimensional model of the track platform is constructed in the digital twin space; The physical operation actions of the equipment on the track phenotype platform are decomposed to determine the corresponding action nodes, and the action nodes are integrated into the three-dimensional model of the track platform to obtain a dynamic simulation model, wherein the physical operation actions of the equipment include: longitudinal y-axis translation of the track, lateral x-axis translation of the track, and z-axis lifting and lowering movement; The sensor equipment mounted on the track phenotyping platform is reconstructed in three dimensions to construct a multi-source sensor imaging unit twin; In the digital twin space, the multi-source sensor imaging unit twin is assembled into the dynamic simulation model to obtain the track phenotype platform twin.

[0007] In some embodiments, the preprocessing of the multi-source phenotypic data to obtain the three-dimensional point cloud data of the planting plot area and the plant phenotypic preprocessing data includes: Point cloud splicing is performed on a plurality of radar point cloud data in the multi-source phenotyping data to obtain regional point cloud model data of the planting plot area; Denoising the regional point cloud model data, and downsampling the denoised regional point cloud model data to obtain three-dimensional point cloud data of the planting plot area; Feature point matching is performed on a plurality of plant image data in the multi-source phenotypic data, and the plant image data with feature point matching is spliced ​​to construct plant phenotypic preprocessing data.

[0008] In some embodiments, determining the spatial coordinates of individual plants in the planting plot area in the digital twin space according to the plant phenotypic preprocessing data includes: Determine the image coordinates of the plant image data in the plant phenotyping preprocessing data in the track phenotyping platform coordinate system; Determining pixel coordinates of a single plant in the plant image data in a track phenotyping platform coordinate system according to the image coordinates; The pixel coordinates are converted into a virtual position coordinate system to obtain spatial coordinates in the digital twin space.

[0009] In some embodiments, determining the pixel coordinates of a single plant in the plant image data in the track phenotyping platform coordinate system according to the image coordinates includes: Determine the real coordinates of the image coordinates in the orbital phenotyping platform coordinate system; Determine the pixel coordinates of any pixel point of a single plant in the plant image data, and determine the pixel difference between the image coordinates and the pixel coordinates; The pixel difference is proportionally corrected by using a reference correction proportional parameter, and the real coordinate is coordinate-corrected by using the corrected pixel difference to obtain the pixel coordinates of the single plant in the plant image data in the track phenotyping platform coordinate system.

[0010] In some embodiments, converting the pixel coordinates into a virtual position coordinate system to obtain spatial coordinates in the digital twin space includes: According to the linear mapping coefficients between the orbital phenotype platform coordinate system and the twin orbital coordinate system, the pixel coordinates are mapped into virtual coordinates in the twin orbital coordinate system; The virtual coordinates are calibrated through the coordinates of the operation starting point of the track phenotyping platform in the twin world coordinate system to obtain the spatial coordinates of the pixel coordinates in the digital twin space.

[0011] In some embodiments, the plant phenotypic preprocessing data includes spectral data and point cloud data, and the multi-source data fusion of the plant phenotypic preprocessing data to obtain the multi-source fusion data includes: Acquire calibrated position information of spectral data and point cloud data, and perform spatial position registration on the spectral data and the point cloud data according to the calibrated position information; The spectral data after spatial position registration is fused based on pixel value weighted calculation to obtain a fused pixel image; Superimposing the fused pixel image with the point cloud data after spatial position registration to obtain multi-source fused data; The step of constructing a plant grid model based on the multi-source fusion data includes: Performing spatial point cloud clustering on the multi-source fusion data to obtain point cloud data of each plant; A grid model is constructed for the point cloud data of each individual plant to obtain a plant grid model.

[0012] In some embodiments, the method further comprises: Acquiring plant phenotypic index samples and plant physiological index samples, and associating the plant phenotypic index samples and the plant physiological index samples into sample pairs; Training an initial deep learning model based on the sample pairs to obtain a physiological index extraction model; Extracting physiological indicators from the spectral indicator data included in the phenotypic indicators through the physiological indicator extraction model to obtain plant physiological indicator prediction data; Mapping the plant physiological index prediction data into a corresponding spatial numerical matrix as data layer physiological index data; The plant physiological index prediction data is structured to obtain physiological index data of multiple plant parts as application layer physiological index data; The data layer physiological indicator data and the application layer physiological indicator data are superimposed on the plant twin.

[0013] In some embodiments, constructing a regional environment twin of the planting plot area includes: Perform three-dimensional reconstruction of environmental sensors other than the track phenotyping platform and ground environment objects other than the plants in the planting plot area to obtain a twin base; Acquire the environmental parameters collected in real time by the environmental sensor and the spatial information of the ground object in the planting plot area; The environmental parameters and the spatial information of the ground objects are integrated into the twin base to obtain a regional environmental twin.

[0014] The present invention also provides a digital twin map construction device for a track-type high-throughput crop phenotyping platform, which is implemented by a digital twin system, wherein the digital twin system comprises: an Internet of Things control module, a data storage module, and a computing service module; The IoT control module is used to obtain multi-source phenotypic data of the planting plot area collected by the track phenotyping platform, and distribute the multi-source phenotypic data to the data storage module and the computing service module; The data storage module is used to store the multi-source phenotypic data distributed by the IoT control module using an object-relational database; The computing service module is used to construct a digital twin map based on the multi-source phenotypic data distributed by the IoT control module or the multi-source phenotypic data stored by the data storage module.

[0015] The method and device for constructing a digital twin map of a track-type high-throughput crop phenotyping platform provided by the present invention collect data of the track phenotyping platform, plants, planting plots, and regional environments in the real environment, and respectively construct corresponding twins in a virtual digital twin space to form a digital twin map of the plant planting scene. In this way, in a unified digital twin space, the fusion application of the multimodal data collected by the phenotyping platform and the regional environmental data of the planting plot area realizes real-time monitoring of the plant planting environment in a virtual space. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a schematic flow chart of the method for constructing a digital twin map of a track-type high-throughput crop phenotyping platform provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the construction of the track phenotype platform twin provided by the present invention.

[0019] Figure 3 It is a schematic diagram of the construction of plant twins provided by the present invention.

[0020] Figure 4 It is a pixel difference schematic diagram of the plant image data provided by the present invention.

[0021] Figure 5 This is a schematic diagram of plant pixel coordinate correction provided by the present invention.

[0022] Figure 6 It is a schematic diagram of the mapping relationship between the real coordinate system and the virtual coordinate system provided by the present invention.

[0023] Figure 7 It is an application schematic diagram of the digital twin system provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figures 1 to 7 The present invention describes a method and device for constructing a digital twin map of a track-based high-throughput crop phenotyping platform. Figure 1 is a flow chart of a method for constructing a digital twin map of a track-type high-throughput crop phenotyping platform provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 106.

[0026] Step 101: Collect multi-source phenotypic data of the planting plot area based on the constructed track phenotyping platform, and construct a track phenotyping platform twin in the digital twin space.

[0027] First, the construction of the track phenotyping platform is introduced. The slide rail of the track phenotyping platform is equipped with a sensor array, which is driven by three motors. The transmission structure provides power to the movement in the x, y, and z directions respectively, and can accurately record the position of the track in the three-axis direction. The sensor array is composed of a variety of phenotypic sensors, which are integrated in an equipment detection box in the form of an array combination, and can collect multi-source phenotypic data through work. Specifically, it includes RGB industrial cameras, multispectral cameras, hyperspectral cameras, thermal infrared cameras, depth cameras, lidar and other sensor equipment.

[0028] The track phenotyping platform is built on the planting plot area. The sensor array can collect multi-source phenotyping data of the planting plot area during the movement of the slide rail. The multi-source phenotyping data has many data types and various data formats. In the embodiment of the present invention, the multi-source phenotyping data is roughly divided into image data, spectral data, and point cloud data. In some implementation scenarios, image data is also a kind of spectral data, which is manifested by different channels. Therefore, the multi-source phenotyping data can also be divided into spectral data and point cloud data only. Among them, image data and spectral data are orthophotos and spectral images obtained by the camera in the sensor array from the top of the plot and the plant through a positive viewing angle. Among them, the orthophoto is a continuous image obtained at a fixed frame rate. A single image reflects the pixel information within the camera's field of view. The data of the entire plot requires algorithmic splicing of the image to form a large plot map that reflects the overall information of the plot. The format of point cloud data is point data distributed in track running strips, which cannot directly reconstruct the three-dimensional morphology and extract phenotypic information. Algorithm reconstruction is required to generate a three-dimensional model. Multiple strips of point cloud data are spliced ​​and reconstructed to generate three-dimensional point cloud data for the entire plot.

[0029] When constructing the digital twin map, it is necessary to construct the track phenotype platform twin in the digital twin space based on the track phenotype platform. Figure 2 , first according to the track phenotyping platform (i.e. Figure 2 The physical structure of the real phenotype platform in the digital twin space is constructed to build a three-dimensional model of the track platform (i.e. Figure 2 Here, the real physical structure of the track phenotyping platform can be accurately mapped, and then the computer can be used to reconstruct the three-dimensional model of the physical form based on the real physical structure. For example, using the construction drawing parameters of the track phenotyping platform, the modeling software can be used to accurately model the three-dimensional model of the track phenotyping platform at a one-to-one ratio, including other buildings and terrain in the operating area involved in the platform.

[0030] However, since the track phenotyping platform needs to perform corresponding actions when performing physical operations, it drives the sensor array to move, thereby collecting multi-source phenotyping data. In order to simulate the physical operation actions of the equipment on the track phenotyping platform, the embodiment of the present invention decomposes the physical operation actions of the equipment on the track phenotyping platform, determines the corresponding action nodes, and integrates the action nodes into the three-dimensional model of the track platform to obtain a dynamic simulation model. The physical operation actions of the equipment on the track phenotyping platform are generally movements in the three-axis directions of the track. Specifically including: longitudinal y-axis translation of the track; lateral x-axis translation of the track; z-axis lifting and lowering movement.

[0031] Here, the three-dimensional model of the track platform can be imported into the simulation platform for development. According to the three decomposed action nodes, the corresponding motion simulation code is input to transform the static three-dimensional model of the track platform into a simulation dynamic model that can be driven by acceptable data. Real-time data transmission is required during the simulation process so that the virtual phenotype platform and the real phenotype platform can follow each other.

[0032] Considering that the track phenotyping platform is equipped with a sensor array, this sensor array is composed of a variety of sensor devices, which are loaded and protected by a box. The sensor array includes a variety of sensor devices such as RGB industrial cameras, multispectral cameras, hyperspectral cameras, thermal infrared cameras, depth cameras, and lidar. When constructing the track phenotyping platform twin, these sensor devices also need to be accurately three-dimensionally modeled. Therefore, the sensor devices loaded on the track phenotyping platform are also three-dimensionally reconstructed here to construct a multi-source sensor imaging unit twin. In the digital twin space, the multi-source sensor imaging unit twin is assembled into the dynamic simulation model to obtain the track phenotyping platform twin.

[0033] Step 102: pre-process the multi-source phenotypic data to obtain three-dimensional point cloud data of the planting plot area and plant phenotypic pre-processed data.

[0034] Here, the multi-source phenotypic data collected by the track phenotyping platform belongs to multi-modal, multi-source phenotypic time series big data. There are many data types and different data formats. If key feature data is to be extracted from a large amount of data, specific preprocessing is required. Therefore, the embodiment of the present invention preprocesses the multi-source phenotypic data here to obtain the three-dimensional point cloud data of the planting plot area and the plant phenotypic preprocessing data.

[0035] Specifically, point cloud splicing is performed on multiple radar point cloud data in the multi-source phenotyping data to obtain regional point cloud model data of the planting plot area. The radar point cloud data here is collected by a lidar, but the lidar is deployed in a sensor array for movement, so the collected radar point cloud data all belong to the local point cloud of the planting plot area. Here, multiple radar point cloud data are spliced ​​to obtain regional point cloud model data of the planting plot area. Point cloud splicing can use an iterative closest point (ICP) algorithm, a SLAM algorithm, or other point cloud splicing algorithms.

[0036] Next, the regional point cloud model data is denoised, and the denoised regional point cloud model data is downsampled to obtain the three-dimensional point cloud data of the planting area. Point cloud denoising can use various filtering algorithms, and point cloud downsampling is to remove redundant point clouds to reduce the amount of calculation. Here, random downsampling or uniform downsampling methods can be used, depending on the specific data volume.

[0037] The RGB industrial camera in the sensor array will collect plant image data in the planting area during movement, which is one of the image data in the multi-source phenotypic data. This is an orthophoto obtained from the top of the plot and the plant. The orthophoto is a continuous image obtained at a fixed frame rate. A single image reflects the pixel information within the camera's field of view. The data of the entire plot needs to be spliced ​​by algorithm to form a large plot map that reflects the overall information of the plot. Therefore, the embodiment of the present invention performs feature point matching on multiple plant image data in the multi-source phenotypic data, and splices the plant image data with matched feature points to construct plant phenotypic preprocessing data.

[0038] Specifically, first detect the feature points of each plant image data, and the detection method can use the SIFT algorithm. Then match the feature points. The matching method can use clustering or calculate similarity, and then determine the similarity transformation matrix between the feature points, align multiple plant image data, and obtain an overall plant image data. The preprocessing process of spectral data is similar to that of plant image data, which will not be repeated here. In addition, when each plant image data is collected, the corresponding point cloud data will also be collected, that is, plant point cloud data. This plant point cloud data is a mixed point cloud of multiple plants, which will be clustered and segmented in the subsequent process for building a plant network model. The mixed point cloud is not processed during the preprocessing process here. Therefore, the overall plant image data and the mixed point cloud can be used as plant phenotypic preprocessing data.

[0039] Step 103: Determine the spatial coordinates of individual plants in the planting plot area in the digital twin space based on the plant phenotypic preprocessing data, perform three-dimensional reconstruction of the three-dimensional point cloud data to obtain a cell map model of the planting plot area, and map the spatial coordinates and the cell map model to the digital twin space to obtain a twin space model.

[0040] The construction of plant twins requires information such as the spatial location of the plants in the planting area, the plot map model of the planting area, and phenotypic indicators. Based on this information, the plants can be mapped in the digital twin space and the digital twin can be constructed.

[0041] like Figure 3 As shown, the construction process of the plant twin is divided into three parts. The first part is the plant coordinate extraction, that is, determining the spatial position of the plant. Therefore, the embodiment of the present invention targets the plant phenotypic preprocessing data and realizes the virtual space plant positioning through coordinate conversion and registration. Here, the spatial coordinates of individual plants in the planting plot area in the digital twin space are determined based on the plant phenotypic preprocessing data, which is described in detail below.

[0042] Firstly, the image coordinates of the plant image data in the plant phenotyping preprocessing data and the image coordinates in the track phenotyping platform coordinate system are determined.

[0043] like Figure 4 As shown, the plant image data refers to the orthophoto (size 4096×3000) obtained by the track phenotyping platform from the top of the plot and the plant through a positive perspective. This orthophoto is the plant image data, which is used as the plant phenotyping preprocessing data after feature point matching and splicing. Since the track phenotyping platform has its own coordinate system, the coordinate information on this coordinate system is obtained through precise positioning of the torque speed sensor on the mobile platform. Therefore, the collected plant image data is accompanied by the coordinate information of the track, which is usually recorded in the form of a file name as x, y coordinates (the pixel coordinate origin is (0, 0), denoted as M, and expressed as ( ). Thus, the image coordinates of the plant image data in the plant phenotyping preprocessing data in the track phenotyping platform coordinate system can be determined. This image coordinate M is generally located at the center point of the plant image data.

[0044] Next, according to the image coordinates, the pixel coordinates of the plant in the plant image data in the track phenotyping platform coordinate system are determined. For a single plant image data, the specific position of the plant can be located by an artificial intelligence algorithm, and the corresponding pixel coordinates can be obtained.

[0045] like Figure 5 As shown, first, the real coordinates of the image coordinates M in the orbital phenotype platform coordinate system can be determined based on the image positioning , denoted as ( ), this real coordinate is based on the real coordinate system of the real coordinate origin ( ). Then the pixel coordinate A of any pixel point of a single plant in the plant image data is determined in the pixel coordinate system (x, y), expressed as ( ), and then determine the pixel difference between the image coordinate M and the pixel coordinate A. This pixel difference is the difference between the image coordinate and the pixel coordinate. Figure 4 As shown in , ) is determined in the pixel coordinate system (x, y) and is used as a reference for the real coordinates in the orbital phenotyping platform coordinate system. Perform coordinate correction.

[0046] Although the mapping relationship between the plant's real position coordinates and the image pixel coordinates is a linear relationship, it cannot be directly mapped and needs to be corrected by a reference object. Therefore, the scale parameter is corrected by a reference object. Pixel difference ( , ) to correct the ratio, and then use the corrected pixel difference to correct the real coordinates Perform coordinate correction to obtain the pixel coordinates of the plant in the plant image data in the track phenotyping platform coordinate system ( ), The calculation process can be expressed by the following formulas (1) to (4): (1)

[0047] (2)

[0048] (3)

[0049] (4)

[0050] In the above formulas (1) to (4), ( ) represents the image coordinate M of the i-th plant image data, ( ) represents the real coordinates of the i-th plant image data in the track phenotyping platform coordinate system , ( ) represents the pixel coordinate A of any pixel point of a single plant in the plant image data, and Represents the pixel difference between the image coordinate M and the pixel coordinate A of any point in the x-axis direction and the y-axis direction respectively.

[0051] After determining the pixel coordinates of the plant in the plant image data in the track phenotyping platform coordinate system, the plant coordinate extraction process is completed, and then the coordinate conversion and registration are performed, that is, the pixel coordinates are converted into ( ) to perform virtual position coordinate system conversion to obtain the spatial coordinates in the digital twin space. The virtual position coordinate system conversion here is the conversion from the orbit platform coordinate system to the twin world coordinate system.

[0052] Here, because the pixel coordinates of individual plants in the image ( ) has been mapped to the orbital platform coordinate system. Figure 6 As shown in the figure, the orbital phenotyping platform collects multi-source phenotyping data in the real planting area, so the orbital platform coordinate system is the coordinate system of the real plot, and the pixel coordinates of the individual plants are based on the real plot coordinates (X, Y). However, in the digital twin space, there is also a virtual twin world coordinate system. Because the orbital phenotyping platform twin is reconstructed in proportion, the twin orbit coordinate system is the mapping of the orbital platform coordinate system in the digital twin space. The twin world coordinate system and the twin orbit coordinate system are parent-child relationships, and the pixel coordinate system is the virtual twin world coordinate system. The loading positioning in the twin space uses the twin world coordinate system, so the pixel coordinates Two more coordinate transformations are required to complete the positioning of the plants in the virtual space.

[0053] The first coordinate transformation is to transform the pixel coordinates ( ), mapped to virtual coordinates in the twin orbit coordinate system. Then the second coordinate transformation is performed, through the coordinates of the operation starting point of the orbital phenotype platform in the twin world coordinate system ( ), calibrate the virtual coordinates in the twin orbit coordinate system and obtain the pixel coordinates ( ) in the digital twin space (i.e., the twin world coordinate system) ( ), which can be expressed as the following formula (5) and formula (6): (5)

[0054] (6)

[0055] Through the above steps, the spatial position of each plant in the plant phenotypic data can be mapped to the digital twin space. Thus, the spatial position of each plant in the digital twin space can be determined, which is convenient for the subsequent construction of plant twins.

[0056] like Figure 3As shown, the second part of the construction process of the plant twin is three-dimensional reconstruction analysis, that is, to reconstruct the cell map model based on the three-dimensional point cloud data of the planting plot area. Here, the three-dimensional point cloud data can be image point cloud data based on plant image data, or it can be depth point cloud data collected by a depth camera, or it can be radar point cloud data collected by a laser radar. However, the image point cloud data is not accurate enough, and the depth point cloud data will also affect the reconstruction accuracy when the outdoor light is unstable. Therefore, the embodiment of the present invention uses radar point cloud data for reconstruction. However, the format of the radar point cloud data is point data distributed in the track running strips. Through laser radar collection, it is impossible to directly reconstruct the three-dimensional morphology and extract phenotypic information. Algorithm reconstruction is required to generate a three-dimensional model. The point cloud data of multiple strips are spliced ​​and reconstructed to generate three-dimensional point cloud data of the entire plot, and then the three-dimensional point cloud data is three-dimensionally reconstructed to obtain a three-dimensional model of the planting plot area, so as to realize the visualization of three-dimensional phenotypic indicators.

[0057] Therefore, the embodiment of the present invention pre-processes the radar point cloud data in the multi-source phenotypic data, obtains the three-dimensional point cloud data of the planting plot area, and then performs three-dimensional reconstruction on the three-dimensional point cloud data to obtain a cell map model of the planting plot area. The three-dimensional point cloud data here is obtained by splicing multiple local point clouds, then denoising and downsampling, and can reflect the panoramic information of the entire planting plot area. Therefore, the three-dimensional point cloud data can be three-dimensionally reconstructed to obtain the corresponding regional terrain and a complete cell map model.

[0058] Next, the spatial coordinates of the individual plants in the digital twin space and the community map model are mapped to the digital twin space to obtain the twin space model as the plant twin base.

[0059] Step 104: perform multi-source data fusion on the plant phenotype preprocessing data to obtain multi-source fusion data, and construct a plant grid model based on the multi-source fusion data.

[0060] like Figure 3 As shown, the third part of the plant twin construction process is phenotypic index extraction, that is, extracting the plant's phenotypic indicators from the plant phenotypic preprocessing data, and finally completing the construction of the plant twin. However, since the plant phenotypic preprocessing data still has the characteristics of large scale, multi-source, and different structures, the extraction of phenotypic indicators such as plant leaf area, biomass, and dry matter will produce a variety of extraction schemes according to different data sources. Although the plant phenotypic preprocessing data has been divided into image data, spectral data, and point cloud data, each type of data can be used to extract performance indicators through a specific scheme, but the data results of a single method are not accurate enough. In order to improve the accuracy of plant phenotypic index extraction, the embodiment of the present invention here performs multi-source data fusion on the plant phenotypic preprocessing data to obtain multi-source fusion data, which is explained in detail below.

[0061] In some embodiments, image data can be used as a type of spectral data, both of which have pixels and resolution, but the difference lies in the channels. Therefore, when multi-source data is fused here, the plant phenotyping preprocessing data is only divided into spectral data and point cloud data for processing. The first is the calibration and registration of the data, and the calibration position information of the spectral data and the point cloud data is obtained. Here, when the track phenotyping platform collects multi-source phenotyping data, a calibration plate is placed under the platform to obtain the calibration position information of image data, spectral data, and point cloud data, which can be used as a reference for position information. Therefore, the spectral data and point cloud data are spatially registered according to the calibration position information. Spatial position registration can use a spatial matrix algorithm to calculate spectral data and point cloud data, wherein the point cloud data is registered with the corresponding calibration position information through the spatial matrix, and the spectral data is registered with the corresponding calibration position information. Because the calibration position information of spectral data is two-dimensional and the calibration position information of point cloud data is three-dimensional, it is necessary to use the spatial matrix to perform transformation calculations during alignment, so that the spatial position and resolution of spectral data and point cloud data are ultimately consistent.

[0062] Next, the spectral data (including image data) after spatial position registration is fused based on pixel value weighted calculation to obtain a fused pixel image. Here, pixel value weighted calculation is also the similarity weighted average calculation used in image fusion. The purpose of fusion processing is to integrate spectral data acquired at different times and in different bands. Since the spectral data has been spatially registered in the previous step, the spatial positions of all spectral data are consistent, and weighted average calculation can be performed directly.

[0063] For example, for any two spectral data that need to be fused, first assign corresponding pixel weights to the two according to image indicators such as the signal-to-noise ratio and clarity of the spectral data, then multiply each pixel value in the spectral data by the corresponding assigned weight, and finally add the pixel values ​​of the two spectral data to find the average value, and finally get a fused pixel image.

[0064] Finally, the fused pixel image is superimposed with the point cloud data after spatial position registration to obtain multi-source fusion data. Here, because the spatial positions of the spectral data and the point cloud data are also the same, the fused pixel image and point cloud data at the same spatial position can be directly superimposed to obtain multi-source fusion data. Therefore, in terms of data composition, this multi-source fusion data is essentially the data of point cloud spatial information and spectral color texture information superimposed together. After the multi-source fusion image is generated, it still needs to be optimized and enhanced, that is, preprocessing including filtering, denoising, downsampling and other steps is performed again.

[0065] After preprocessing the multi-source fusion data, the plant grid model is constructed based on the multi-source fusion data. Here, because the plant phenotype preprocessing data actually reflects the overall information of the plants in the planting area, the multi-source fusion data here can also reflect the index data of each plant in the planting area. When constructing the plant twin, it is necessary to segment the index data of each plant. The index data here refers to the point cloud data of a single plant.

[0066] Therefore, the embodiment of the present invention first performs spatial point cloud clustering on the multi-source fusion data to obtain the point cloud data of each individual plant. By calling the point cloud clustering algorithm, the point cloud data included in the diverse fusion data is clustered, thereby identifying and segmenting the point cloud data of each individual plant in the planting area through clustering. Next, a deep learning model that fuses the point cloud data can be trained to construct a grid model for the point cloud data of each individual plant to obtain a plant grid model. The deep learning model here can complete the missing point cloud data of the individual plant, and then perform three-dimensional reconstruction to construct a grid model, and reconstruct the point cloud data of the individual plant into a corresponding plant grid model. This plant grid module can be directly loaded into the digital twin space for constructing a plant twin.

[0067] Step 105: Analyze the multi-source fusion data to obtain phenotypic indicators, and superimpose the phenotypic indicators and the plant grid model on the twin space model to obtain the plant twin.

[0068] As mentioned above, multi-source data fusion is performed on plant phenotypic preprocessing data in order to improve the accuracy of plant phenotypic index extraction. The phenotypic indicators of plants belong to multimodal data information, and these data information are the parameters required to render plant twins in the digital twin space. Therefore, the embodiment of the present invention also parses the multi-source fusion data to obtain phenotypic indicators. Here, based on the multi-source fusion data, the phenotypic analysis algorithm tool can be called to perform direct analysis and processing to obtain a series of phenotypic indicators of the plant, such as plant color, vegetation index, shape, texture, plant height, volume, etc. These phenotypic indicators need to be superimposed on the corresponding plant twins.

[0069] After the phenotypic indicators are obtained through analysis, the phenotypic indicators and the plant grid model are superimposed on the twin space model to obtain the plant twin. Through step 103, the spatial coordinates of the individual plant and the cell map model have been mapped to the digital twin space to obtain the twin space model as the plant twin base. And through step 104, the plant grid model of the individual plant is constructed based on the multi-source fusion data. Therefore, in the digital twin space, when the data of the plant twin base is superimposed, the plant grid model of the individual plant can be placed one by one at the corresponding spatial coordinates according to the spatial coordinates of each plant in the twin space model, and then the data information corresponding to the phenotypic indicators is also added to the corresponding plant grid model, and finally the corresponding cell map model is combined for overall rendering to finally construct the plant twin.

[0070] In some embodiments, in order to make the constructed plant twins have physiological information to characterize the physiological conditions of the plants, including photosynthesis, plant metabolism, plant respiration, plant water physiology, plant mineral nutrition, plant internal transport, growth and development. Therefore, in order to facilitate planting management, physiological information also needs to be integrated into the construction of plant twins. The physiological information extraction comes from the phenotypic indicators of the plants, that is, plant color, vegetation index, shape, texture, plant height, volume and other data. However, the phenotypic indicator data belongs to multimodal data, so the spectral indicator data included in the phenotypic indicators is used here to extract physiological indicators. The extraction process can be implemented using a physiological indicator extraction model.

[0071] First, a deep learning module needs to be trained to obtain a physiological indicator extraction model. Here, plant phenotypic indicator samples and plant physiological indicator samples are obtained, and the plant phenotypic indicator samples and plant physiological indicator samples are associated into sample pairs. By collecting a large number of plant phenotypic indicators and physiological indicators as samples, a mapping relationship between plant phenotypic indicators and physiological indicators can be established based on prior knowledge. Then, based on this mapping relationship, the plant phenotypic indicator samples and plant physiological indicator samples are associated into sample pairs, and then the initial deep learning model is trained based on the sample pairs to obtain the physiological indicator extraction model. During the training process, the deep learning model is allowed to fully learn the association characteristics between phenotypic indicators and physiological indicators, so that the trained physiological indicator extraction model can extract the corresponding physiological indicator data based on the phenotypic indicator data.

[0072] After the training of the physiological index extraction model is completed, the physiological index extraction model can be used to extract physiological indexes from the spectral index data included in the phenotypic index to obtain plant physiological index prediction data. The plant physiological index prediction data can be used to integrate with the plant twins to add physiological information to the plant twins.

[0073] However, the fusion process needs to be divided into fusion on the data layer and fusion on the application layer. Specifically, on the data layer, the plant physiological index prediction data is mapped to the corresponding spatial numerical matrix as the data layer physiological index data. Here, the plant physiological index prediction data is converted into a spatial numerical matrix, such as a heat map that can reflect the plant physiological index, as the data layer physiological index data.

[0074] As for the application layer, the plant physiological index prediction data is structured to obtain multiple plant part physiological index data as the application layer physiological index data. The physiological indicators of the application layer generally include plant height, plant biomass, dry matter content, etc. These data are one-dimensional parameter indicators, and these physiological index parameters will change. For example, with the growth of time, temperature changes, and climate changes, the plant height, plant biomass, and dry matter content of the plant are constantly changing, and the changes in the physiological index data of each part of each plant are different. Therefore, in order to facilitate the superposition of real-time changing physiological index data on plant twins. Here, the plant physiological index prediction data is structured to obtain multiple plant part physiological index data. That is, the structure is divided according to the parts of the plant, and the physiological index data of the corresponding plant parts are extracted respectively.

[0075] Finally, the data layer physiological indicator data and the application layer physiological indicator data are superimposed on the plant twin. Here, for the data layer physiological indicator data, such as the heat map of the plant physiological indicators, the heat map can be directly fused with the plant twin, so as to add physiological information to the plant twin. For the application layer physiological indicator data, the physiological indicator data of the corresponding plant part can be loaded into the plant twin for rendering. When loading new physiological indicator data, it is only necessary to query the old physiological indicator data of the corresponding part and replace the old physiological indicator data of the corresponding part with the new physiological indicator data, so as to realize the real-time rendering of the physiological information of the plant twin.

[0076] Step 106: construct a regional environmental twin of the planting plot area, and integrate the track phenotype platform twin, the plant twin, and the regional environment twin in the digital twin space to obtain a digital twin map.

[0077] During the construction of the digital twin map, after mapping the track phenotyping platform and the plants in the planting area to the digital twin space, considering that the planting area may also involve other environmental perception facilities, these facilities also provide environmental parameters for the planting of plants, so it is also necessary to construct a twin. Here, the regional environment is also reconstructed in three dimensions, the corresponding twin base is constructed, and the acquired real-time environmental parameters are connected to form a regional environmental twin. In addition, there are other land environment objects. Such objects can directly reflect the spatial information of the operation scene in the planting area, and can be reconstructed in a three-dimensional model in a one-to-one ratio in the twin space. The following is a detailed explanation.

[0078] First, the environmental sensors other than the track phenotyping platform and the ground environment objects other than the plants in the planting plot area are three-dimensionally reconstructed to obtain a twin base. In the planting plot area, in addition to the track phenotyping platform, there are also a variety of environmental sensors such as temperature, humidity, and light, which are used to monitor the temperature, humidity, and light of plant growth in real time. In addition, in addition to plants, other ground environment objects (such as plant markers, weeds, auxiliary growth equipment, etc.) are also involved, and these ground environment objects also have corresponding ground space information. Therefore, these environmental sensors and ground environment objects also need to be three-dimensionally reconstructed and mapped to the digital twin space to establish a twin base. Therefore, when constructing the regional environmental twin here, the environmental parameters collected in real time by the environmental sensors and the ground space information of the ground environment objects in the planting plot area are obtained, and finally the environmental parameters and ground space information are integrated into the twin base to obtain the regional environmental twin.

[0079] Finally, the track phenotyping platform twin, plant twin, and regional environment twin are integrated in the digital twin space, and finally a digital twin map of the planting plot area is obtained. In this way, the plant body conditions in the planting plot area in the real world can be accurately simulated in the virtual digital twin space, and a continuous and monitorable dynamic digital twin display of the plant body in the planting plot area can be achieved. The defect of the lack of digital twin construction process for the plant body in the process of constructing the plant phenotype digital twin is overcome. In addition, the multimodal data collected by the track phenotyping platform are integrated into the construction process of the plant twin one by one, which also overcomes the defect of the multimodal data collected on the phenotyping platform and the lack of fusion application in a unified twin space.

[0080] In order to provide data support and algorithm support for the digital twin map construction method in the present invention, an embodiment of the present invention also provides a track-type high-throughput crop phenotyping platform digital twin map construction device, which can actually be implemented through a digital twin system to provide real-time data processing and various algorithm service support for the construction of the digital twin map, which is described in detail below.

[0081] like Figure 7As shown in the figure, the digital twin system specifically includes: IoT control module, data storage module, and algorithm service module. Among them, the IoT control module is used to obtain the multi-source phenotypic data of the planting plot area collected and uploaded by the track phenotyping platform, and distribute the multi-source phenotypic data to the data storage module and the computing service module. The IoT control module specifically manages the data sources (including multi-source phenotypic data) collected from the planting plot area, and is responsible for data transmission between the data storage module and the algorithm service module.

[0082] The data storage module is used to store the multi-source phenotypic data distributed by the IoT control module using an object-relational database. Specifically, the data storage module is used to obtain the multi-source phenotypic data of the planting plot area uploaded in real time through the IoT control module, and adopts a distributed architecture and is designed to use a database that combines an object-relational database with an SQL database to store the multi-source phenotypic data of the planting plot area. The data storage module is actually a data storage center, and the multi-source phenotypic data of the planting plot area stored includes: RGB images, hyperspectral images, multispectral images, lidar point clouds, depth point clouds, thermal infrared images, video streams, etc. These data have different data formats and huge amounts of data. In order to support this multi-source, cross-domain, and multi-modal big data feature, the data center is designed to use an object-relational database combined with an SQL database for data storage to ensure data security.

[0083] The computing service module is used to construct a digital twin map based on the multi-source phenotypic data distributed by the IoT control module or the multi-source phenotypic data stored by the data storage module. Specifically, the algorithm service module can construct a digital twin map based on the multi-source phenotypic data distributed by the IoT control module or the multi-source phenotypic data stored by the data storage module. Figure 1 In the steps 101 to 106 shown in , a digital twin space is designed based on the algorithm, and then the track phenotype platform twin, plant twin and regional environment twin are respectively constructed in the digital twin space according to the multi-source phenotypic data, and the track phenotype platform twin, plant twin and regional environment twin are integrated to obtain a digital twin map. The computing service module can be a computing center deployed in the cloud, which can call data from the IoT control module and the data storage module, and then perform key computing services such as data processing, analysis and calculation and decision-making, receive service requests through standard interfaces, and feedback the calculation results, which involves building a digital twin map based on multi-source phenotypic data as the calculation result. The server for cloud algorithm deployment adopts a distributed cluster, and the algorithm integration adopts standard technology to form a unified and manageable algorithm call interface.

[0084] The method and device for constructing a digital twin map of a track-type high-throughput crop phenotyping platform provided by the present invention accurately reconstruct the track phenotyping platform, monitor real objects such as individual plants, planting plots and regional environments in a virtual digital twin space, and form a digital twin map of the plant planting scene. Thus, in a unified digital twin space, the multimodal data collected by the phenotyping platform and the regional environmental data of the planting plot area are integrated and applied, and the track-type high-throughput crop phenotyping platform digital twin map construction system provides computing support and hardware support for the construction of the data twin map. In addition, various modules are set in the system for human-computer interaction. Plant growth in the planting plot area in the real world is realized, and real-time monitoring and computational simulation of the plant planting environment are performed in the virtual virtual space, and decisions are made on plant growth.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform, characterized in that: The method comprises: Collect multi-source phenotypic data of the planting area based on the constructed track phenotyping platform, and build a track phenotyping platform twin in the digital twin space; Preprocessing the multi-source phenotypic data to obtain three-dimensional point cloud data of the planting plot area and plant phenotypic preprocessing data; Determine the spatial coordinates of individual plants in the planting plot area in the digital twin space according to the plant phenotypic preprocessing data, perform three-dimensional reconstruction on the three-dimensional point cloud data to obtain a cell map model of the planting plot area, and map the spatial coordinates and the cell map model to the digital twin space to obtain a twin space model; Performing multi-source data fusion on the plant phenotype preprocessing data to obtain multi-source fusion data, and constructing a plant grid model based on the multi-source fusion data; Analyzing the multi-source fusion data to obtain phenotypic indicators, and superimposing the phenotypic indicators and the plant grid model on the twin space model to obtain plant twins; A regional environmental twin of the planting plot area is constructed, and the track phenotype platform twin, the plant twin, and the regional environment twin are integrated in the digital twin space to obtain a digital twin map.

2. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 1, characterized in that: The construction of the track phenotype platform twin in the digital twin space includes: According to the physical structure of the track phenotype platform, a three-dimensional model of the track platform is constructed in the digital twin space; The physical operation actions of the equipment on the track phenotype platform are decomposed to determine the corresponding action nodes, and the action nodes are integrated into the three-dimensional model of the track platform to obtain a dynamic simulation model, wherein the physical operation actions of the equipment include: longitudinal y-axis translation of the track, lateral x-axis translation of the track, and z-axis lifting and lowering movement; The sensor equipment mounted on the track phenotyping platform is reconstructed in three dimensions to construct a multi-source sensor imaging unit twin; In the digital twin space, the multi-source sensor imaging unit twin is assembled into the dynamic simulation model to obtain the track phenotype platform twin.

3. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 1, characterized in that: The preprocessing of the multi-source phenotypic data to obtain three-dimensional point cloud data of the planting plot area and plant phenotypic preprocessing data includes: Point cloud splicing is performed on a plurality of radar point cloud data in the multi-source phenotyping data to obtain regional point cloud model data of the planting plot area; Denoising the regional point cloud model data, and downsampling the denoised regional point cloud model data to obtain three-dimensional point cloud data of the planting plot area; Feature point matching is performed on a plurality of plant image data in the multi-source phenotypic data, and the plant image data with feature point matching is spliced ​​to construct plant phenotypic preprocessing data.

4. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 1, characterized in that: Determining the spatial coordinates of individual plants in the planting plot area in the digital twin space according to the plant phenotype preprocessing data includes: Determine the image coordinates of the plant image data in the plant phenotyping preprocessing data in the track phenotyping platform coordinate system; Determining pixel coordinates of a single plant in the plant image data in a track phenotyping platform coordinate system according to the image coordinates; The pixel coordinates are converted into a virtual position coordinate system to obtain the spatial coordinates of the pixel coordinates in the digital twin space.

5. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 4, characterized in that: Determining pixel coordinates of a single plant in the plant image data in a track phenotyping platform coordinate system according to the image coordinates includes: Determine the real coordinates of the image coordinates in the orbital phenotyping platform coordinate system; Determine the pixel coordinates of any pixel point of a single plant in the plant image data, and determine the pixel difference between the image coordinates and the pixel coordinates; The pixel difference is proportionally corrected by using a reference correction proportional parameter, and the real coordinate is coordinate-corrected by using the corrected pixel difference to obtain the pixel coordinates of the single plant in the plant image data in the track phenotyping platform coordinate system.

6. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 4, characterized in that: The converting the pixel coordinates into a virtual position coordinate system to obtain the spatial coordinates of the pixel coordinates in the digital twin space includes: According to the linear mapping coefficients between the orbital phenotype platform coordinate system and the twin orbital coordinate system, the pixel coordinates are mapped into virtual coordinates in the twin orbital coordinate system; The virtual coordinates are calibrated through the coordinates of the operation starting point of the track phenotyping platform in the twin world coordinate system to obtain the spatial coordinates of the pixel coordinates in the digital twin space.

7. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 1, characterized in that: The plant phenotype preprocessing data includes spectral data and point cloud data, and the multi-source data fusion is performed on the plant phenotype preprocessing data to obtain the multi-source fusion data, including: Acquire calibrated position information of spectral data and point cloud data, and perform spatial position registration on the spectral data and the point cloud data according to the calibrated position information; The spectral data after spatial position registration is fused based on pixel value weighted calculation to obtain a fused pixel image; Superimposing the fused pixel image with the point cloud data after spatial position registration to obtain multi-source fused data; The step of constructing a plant grid model based on the multi-source fusion data includes: Performing spatial point cloud clustering on the multi-source fusion data to obtain point cloud data of each plant; A grid model is constructed for the point cloud data of each individual plant to obtain a plant grid model.

8. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 1, characterized in that: The method further comprises: Acquiring plant phenotypic index samples and plant physiological index samples, and associating the plant phenotypic index samples and the plant physiological index samples into sample pairs; Training an initial deep learning model based on the sample pairs to obtain a physiological index extraction model; Extracting physiological indicators from the spectral indicator data included in the phenotypic indicators through the physiological indicator extraction model to obtain plant physiological indicator prediction data; Mapping the plant physiological index prediction data into a corresponding spatial numerical matrix as data layer physiological index data; The plant physiological index prediction data is structured to obtain physiological index data of multiple plant parts as application layer physiological index data; The data layer physiological indicator data and the application layer physiological indicator data are superimposed on the plant twin.

9. The method for constructing a digital twin map of a track-based high-throughput crop phenotyping platform according to claim 1, characterized in that: The construction of the regional environmental twin of the planting plot area includes: Perform three-dimensional reconstruction of environmental sensors other than the track phenotyping platform and ground environment objects other than the plants in the planting plot area to obtain a twin base; Acquire the environmental parameters collected in real time by the environmental sensor and the spatial information of the ground object in the planting plot area; The environmental parameters and the spatial information of the ground objects are integrated into the twin base to obtain a regional environmental twin.

10. A track-type high-throughput crop phenotyping platform digital twin map construction device, characterized in that: The device is implemented by a digital twin system, and the digital twin system includes: Internet of Things control module, data storage module and computing service module; The IoT control module is used to obtain multi-source phenotypic data of the planting plot area collected by the track phenotyping platform, and distribute the multi-source phenotypic data to the data storage module and the computing service module; The data storage module is used to store the multi-source phenotypic data distributed by the IoT control module using an object-relational database; The computing service module is used to construct a digital twin map based on the multi-source phenotypic data distributed by the IoT control module or the multi-source phenotypic data stored by the data storage module.

Images