Field crop three-dimensional reconstruction method, device, equipment and program product

Abstract

The invention discloses a three-dimensional reconstruction method, device, equipment and program product for field crops, and relates to the technical field of agricultural information and computer vision. The method comprises the following steps: acquiring a target single-view image of a target field crop; performing three-dimensional information reasoning on the target single-view-angle image by adopting a three-dimensional reconstruction model to obtain a reference space coordinate and a reference observation direction of a target field scene; determining a reference color and a reference volume density of the target field scene according to the reference space coordinate and the reference observation direction of the target field scene; according to the reference color and the reference volume density of the target field scene, performing three-dimensional reconstruction on the target field crop to obtain a target three-dimensional color point cloud of the target field crop; wherein the target three-dimensional color point cloud comprises a target space coordinate and a target color. According to the technical scheme of the embodiment of the invention, the three-dimensional reconstruction of the field crops based on the single-view image is realized, and the three-dimensional reconstruction efficiency and reconstruction precision of the field crops are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural information technology and computer vision technology, and in particular to a field crop three-dimensional reconstruction method, device, equipment and program product. BACKGROUND

[0002] Crop phenotype analysis is of great significance in breeding and agricultural production. By accurately measuring the morphological characteristics of crops (such as plant height, leaf area, stem thickness, etc.), key data can be provided for variety selection, growth monitoring and yield prediction. However, traditional crop phenotype acquisition methods mainly rely on manual measurement or two-dimensional image analysis, which not only consumes time and effort, but also is easily affected by human error and perspective limitations.

[0003] With the development of computer vision and three-dimensional reconstruction technology, more and more researches have begun to try to use three-dimensional reconstruction means to obtain the fine structure of plants, so as to more comprehensively and accurately extract crop phenotype information. The existing three-dimensional reconstruction method is a traditional multi-view reconstruction method. Usually, multi-view photogrammetry (SfM (Structure from Motion) / MVS (Multi-View Stereo)) technology is used, such as using COLMAP (Comprehensive Camera Model and Structure Light Motion Library) for feature matching and sparse point cloud reconstruction, and then generating dense point cloud through OpenMVS (Multi-View Stereo Reconstruction Soft, an open source technology framework for multi-view stereo reconstruction).

[0004] However, the existing technology often needs to collect a large number of images at different angles when obtaining the three-dimensional model of field crops, and the reconstruction process is time-consuming and sensitive to field scenes (such as light and wind movement). In addition, for crops with complex structure (such as corn plants with a large number of interlaced leaves), the reconstruction is often incomplete or the accuracy is poor. SUMMARY

[0005] The present application provides a field crop three-dimensional reconstruction method, device, equipment and program product, which realizes three-dimensional reconstruction of field crops based on single-view images, and improves the efficiency and accuracy of field crop three-dimensional reconstruction.

[0006] According to an aspect of the present application, a field crop three-dimensional reconstruction method is provided, the method comprising:

[0007] obtaining a target single-view image of a target field crop;

[0008] using a three-dimensional reconstruction model to perform three-dimensional information reasoning on the target single-view image, to obtain a reference spatial coordinate and a reference observation direction of a target field scene;

[0009] determine reference color and reference volume density of the target field scene according to the reference spatial coordinates and the reference observation direction of the target field scene;

[0010] perform three-dimensional reconstruction on the target field crop according to the reference color and the reference volume density of the target field scene to obtain a target three-dimensional color point cloud of the target field crop; wherein the target three-dimensional color point cloud comprises target spatial coordinates and target color.

[0011] According to another aspect of the present application, there is provided a field crop three-dimensional reconstruction device, the device comprising:

[0012] a target monocular image acquisition module configured to acquire a target monocular image of a target field crop;

[0013] a three-dimensional reconstruction model inference module configured to perform three-dimensional information inference on the target monocular image by using a three-dimensional reconstruction model to obtain reference spatial coordinates and a reference observation direction of a target field scene;

[0014] a reference color calculation module configured to determine reference color and reference volume density of the target field scene according to the reference spatial coordinates and the reference observation direction of the target field scene;

[0015] a target three-dimensional color point cloud reconstruction module configured to perform three-dimensional reconstruction on the target field crop according to the reference color and the reference volume density of the target field scene to obtain a target three-dimensional color point cloud of the target field crop; wherein the target three-dimensional color point cloud comprises target spatial coordinates and target color.

[0016] According to another aspect of the present application, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory in communication with the at least one processor; wherein

[0019] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the field crop three-dimensional reconstruction method according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to perform the field crop three-dimensional reconstruction method according to any one of the embodiments of the present application when executed by the processor.

[0021] According to another aspect of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the field crop three-dimensional reconstruction method according to any one of the embodiments of the present application.

[0022] The technical solution of the embodiments of the present application breaks through the limitation of local perception of the prior art, improves the global inference capability based on the target monocular image of the target field crop, and only needs the target monocular image of the target field crop to complete the three-dimensional reconstruction of the target field crop. Compared with the traditional multi-view three-dimensional reconstruction and the three-dimensional reconstruction method using special equipment, the data acquisition difficulty and equipment cost are greatly reduced. In actual application, there is no need to add expensive sensors or build complex image acquisition systems, which has high economic efficiency and convenience. At the same time, the monocular image processing, three-dimensional information model inference and three-dimensional color point cloud generation are seamlessly integrated to form an end-to-end automatic process. The user of the three-dimensional reconstruction does not need to have a computer vision or programming background, and only needs to perform simple operations to obtain the result, which has good usability. Moreover, the end-to-end architecture also reduces the errors of intermediate data transmission and conversion, improves the processing efficiency and stability of the field crop three-dimensional reconstruction. Moreover, unlike only obtaining the spatial structure of the target field crop, the technical solution of the embodiments of the present application outputs the three-dimensional color point cloud of the target field crop, which can realistically reproduce the appearance of the target field crop, facilitate the verification by the human eye and subsequent intuitive analysis. Specifically, by retaining the color features of the target field crop, further pest detection can be facilitated. At the same time, the color texture helps to accurately segment different organs of the target field crop, improves the accuracy of crop phenotype analysis, and thus improves the applicability in the field of crops.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flowchart of a field crop three-dimensional reconstruction method according to an embodiment of the present application;

[0026] Figure 2is a flow chart of a field crop three-dimensional reconstruction method according to Embodiment Two of the present application;

[0027] Figure 3 is a flow chart of a field crop three-dimensional reconstruction method according to Embodiment Three of the present application;

[0028] Figure 4 is a three-dimensional reconstruction result of field corn images of different perspectives according to Embodiment Three of the present application;

[0029] Figure 5 is a structural schematic diagram of a field crop three-dimensional reconstruction device according to Embodiment Four of the present application;

[0030] Figure 6 is a structural schematic diagram of an electronic device implementing a field crop three-dimensional reconstruction method according to Embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without making creative efforts should belong to the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment One

[0034] Figure 1 is a flow chart of a field crop three-dimensional reconstruction method according to Embodiment One of the present application. The embodiments of the present application can be applicable to the case of three-dimensional reconstruction of crops in the field, and the method is executed by a field crop three-dimensional reconstruction device, which is realized in the form of hardware and / or software, and can be configured in an electronic device that carries the field crop three-dimensional reconstruction function.

[0035] Referring to Figure 1 The field crop three-dimensional reconstruction method shown includes:

[0036] S101, acquiring a target monocular image of a target field crop.

[0037] The target field crop is a crop growing in a field scene. The target field crop is the analysis object of crop phenotype analysis. The target monocular image is a single image taken from a single perspective of the target field crop. Exemplarily, the target monocular image is an RGB (Red Green Blue) image. The prior art usually adopts multi-perspective images for three-dimensional reconstruction to realize three-dimensional reconstruction of an object. In comparison, three-dimensional reconstruction using a monocular image can reduce the number of images required for three-dimensional reconstruction, thereby improving the reconstruction efficiency and applicability of three-dimensional reconstruction. However, the prior art cannot realize three-dimensional reconstruction using a monocular image, which can be understood as the prior art can only guarantee the accuracy of three-dimensional reconstruction in the single perspective direction, and the accuracy of three-dimensional reconstruction in other perspective directions is low.

[0038] Optionally, a camera configured by a mobile terminal is used to acquire the target monocular image of the target field crop in real time. Exemplarily, the mobile terminal includes a smartphone, a tablet computer, or a mobile robot, etc. Optionally, a monitoring camera fixedly installed in the field scene is used to periodically acquire the target monocular image of the target field crop.

[0039] Optionally, while acquiring the target monocular image, acquisition information such as acquisition time, acquisition location, and acquisition device parameters is acquired. The acquisition device parameters include focal length and resolution, etc.

[0040] Optionally, after acquiring the target monocular image, the target monocular image is preprocessed, for example, based on the input requirements of the three-dimensional reconstruction model, the target monocular image is scaled and cropped to realize resolution adjustment of the target monocular image. For another example, the target monocular image is color space converted and normalized, and the RGB value of the target monocular image is normalized to the interval [0, 1]. For another example, a filtering or defogging algorithm is used to denoise and enhance the target monocular image to avoid the problem of uneven illumination or motion blur in the field scene, and to improve the quality of the target monocular image.

[0041] S102, using a three-dimensional reconstruction model to perform three-dimensional information reasoning on the target monocular image to obtain a reference spatial coordinate and a reference observation direction of the target field scene.

[0042] The three-dimensional reconstruction model is used for three-dimensional information reasoning of the single-view image. The three-dimensional reconstruction model is a deep learning model. For example, the three-dimensional reconstruction model is a large reconstruction model (LRM) based on deep learning. The target field scene is a spatial scene corresponding to the target single-view image. The reference spatial coordinate is a three-dimensional coordinate of a spatial point in the target field scene. For example, the reference spatial coordinate is represented as (x, y, z). The reference observation direction is an observation direction of a spatial point in the target field scene. For example, the reference observation direction is represented as .

[0043] Specifically, the target single-view image is input into the three-dimensional reconstruction model, and the reference spatial coordinate and the reference observation direction of the target field scene are output.

[0044] S103, according to the reference spatial coordinate and the reference observation direction of the target field scene, the reference color and the reference body density of the target field scene are determined.

[0045] The reference color is used to represent the color intensity of the light emitted or scattered along the reference observation direction at the spatial point corresponding to the reference spatial coordinate in the target field scene. The reference color is used to simulate the visual effect of the target field crop surface. The reference color is used for rendering the target field crop. The reference body density is used to represent the absorption or scattering ability of the light by the spatial point corresponding to the reference spatial coordinate in the target field scene. The reference body density is used to represent the probability density of the light being intercepted by the medium particles at the spatial point corresponding to the reference spatial coordinate in the target field scene. The reference body density is used to extract the three-dimensional structure of the target field crop.

[0046] Specifically, a neural radiation field model is used to infer the reference color and the reference body density of the target field scene based on the reference spatial coordinate and the reference observation direction of the target field scene.

[0047] S104, according to the reference color and the reference body density, the target field crop is three-dimensionally reconstructed to obtain a target three-dimensional color point cloud of the target field crop.

[0048] The target three-dimensional color point cloud is a three-dimensional color point cloud corresponding to the target field crop in the target field scene. The target three-dimensional color point cloud includes a target spatial coordinate and a target color. The target spatial coordinate is the spatial coordinate of each spatial point of the target field crop itself in the target field scene. The target color is the color corresponding to the target spatial coordinate. The target color is used to represent the color intensity of the light emitted or scattered along the reference observation direction at the spatial point corresponding to the target spatial coordinate in the target field scene. The target color is used to simulate the visual effect of the target field crop surface.

[0049] Specifically, the pixel color of the single reference spatial coordinate is determined according to the reference color and the reference volume density of the single reference spatial coordinate by using a volume rendering formula.

[0050] For example, the pixel color of the single reference spatial coordinate is determined according to the reference color and the reference volume density of the single reference spatial coordinate by using a volume rendering formula:

[0051] ;

[0052] In the formula, C(r) is the pixel color of the single reference spatial coordinate; t far is the light ray termination sampling depth; t near is the light ray starting sampling depth; T(t) is the transmittance, representing the probability that the light ray is not blocked from the starting point t near to the current position t. is the reference volume density. is the reference color.

[0053] In the formula, the following formula is used to represent the calculation formula of the transmittance obtained by integrating the volume density along the light ray path:

[0054] ;

[0055] In the formula, T(t) is the transmittance, representing the probability that the light ray is not blocked from the starting point t near to the current position t; t near is the light ray starting sampling depth; t is the current position. is the volume density.

[0056] In an optional embodiment of the present application, the target field crop is three-dimensionally reconstructed according to the reference color and the reference volume density of the target field scene to obtain the target three-dimensional color point cloud of the target field crop, including: for the single reference spatial coordinate, comparing the reference volume density with a preset volume density; for the single reference spatial coordinate, when the reference volume density is greater than the preset volume density, determining that the reference spatial coordinate is a target spatial coordinate; determining the target color corresponding to the target spatial coordinate according to the reference color corresponding to the target spatial coordinate; and three-dimensionally reconstructing the target field crop according to the target colors of the target spatial coordinates to obtain the target three-dimensional color point cloud of the target field crop.

[0057] The preset body density is a preset lower limit value of the body density corresponding to the spatial coordinate of the target field crop itself. The preset body density is used to distinguish the target field crop and the background of the target field scene. When the reference body density is greater than the preset body density, it can be understood that the reference spatial coordinate is the spatial coordinate of the spatial point corresponding to the target field crop itself. When the reference body density is less than or equal to the preset body density, it can be understood that the reference spatial coordinate is the spatial coordinate corresponding to the background of the target field scene.

[0058] Specifically, for a single reference spatial coordinate, the reference body density and the preset body density are compared. When the reference body density is greater than the preset body density, the reference spatial coordinate is determined as the target spatial coordinate corresponding to the target field crop. The reference color corresponding to the target spatial coordinate is determined as the target color corresponding to the target spatial coordinate. The target field crop is three-dimensionally reconstructed according to the target colors of the target spatial coordinates, and a target three-dimensional color point cloud of the target field crop is obtained.

[0059] The scheme compares the reference body density and the preset body density, and determines the reference spatial coordinate as the target spatial coordinate when the reference body density is greater than the preset body density, thereby improving the determination efficiency of the spatial structure of the target field crop. The target color corresponding to the target spatial coordinate is determined according to the reference color corresponding to the target spatial coordinate, and the target field crop is three-dimensionally reconstructed according to the target colors of the target spatial coordinates, thereby obtaining a target three-dimensional color point cloud of the target field crop, and further improving the three-dimensional reconstruction efficiency of the field crop.

[0060] The technical scheme of the embodiment of the present application breaks through the limitation of local perception of the prior art, improves the global inference capability based on the target monocular image of the field crop, and only needs the target monocular image of the field crop, so that the three-dimensional reconstruction of the target field crop can be completed. Compared with the traditional multi-view three-dimensional reconstruction and the three-dimensional reconstruction method using special equipment, the data acquisition difficulty and equipment cost are greatly reduced. In actual application, there is no need to add expensive sensors or build a complex image acquisition system, which has high economic efficiency and convenience. At the same time, the monocular image processing, three-dimensional information model inference and three-dimensional color point cloud generation are seamlessly integrated to form an end-to-end automatic process. The user of three-dimensional reconstruction does not need to have a computer vision or programming background, and only needs to perform simple operations to obtain the result, which has good usability. Moreover, the end-to-end architecture also reduces the errors of intermediate data transmission and conversion, improves the processing efficiency and stability of the three-dimensional reconstruction of the field crop. Furthermore, unlike only obtaining the spatial structure of the target field crop, the technical scheme of the embodiment of the present application outputs the three-dimensional color point cloud of the target field crop, which can realistically reproduce the appearance of the target field crop, and is convenient for the human eye to verify and subsequent intuitive analysis. Specifically, by retaining the color characteristics of the target field crop, further disease and pest detection can be facilitated. At the same time, the color texture helps to accurately segment different organs of the target field crop (such as distinguishing leaves and stems), which improves the accuracy of crop phenotype analysis, thereby improving the applicability in the field of crops.

[0061] In an optional embodiment of the present application, after obtaining the target monocular image of the target field crop, the method further comprises: performing foreground segmentation on the target monocular image to update the target monocular image. Correspondingly, after obtaining the target three-dimensional color point cloud of the target field crop, the method further comprises: performing foreground segmentation on the target three-dimensional color point cloud to update the target three-dimensional color point cloud.

[0062] Specifically, a first visual object segmentation model is used to perform foreground segmentation on the target monocular image, and an image mask segmentation result of the target field crop is used to update the target monocular image. After obtaining the target three-dimensional color point cloud of the target field crop, a second visual object segmentation model is used to perform foreground segmentation on the target three-dimensional color point cloud, and a three-dimensional mask segmentation result of the target field crop is used to update the target three-dimensional color point cloud. The first visual object segmentation model is used to segment the target field crop from the target monocular image to obtain the image mask segmentation result of the target field crop. The image mask segmentation result of the target field crop is the segmentation result of the target field crop in the target monocular image. The second visual object segmentation model is used to segment the target field crop from the target three-dimensional color point cloud to obtain the three-dimensional mask segmentation result of the target field crop. The three-dimensional mask segmentation result of the target field crop is the segmentation result of the target field crop in the target three-dimensional color point cloud. The first visual object segmentation model and the second visual object segmentation model may include a convolutional neural network model, a deep learning model, or a reinforcement learning model, etc.

[0063] The present scheme avoids the interference of background clutter in the field environment on three-dimensional reconstruction by performing foreground segmentation on the target monocular image after obtaining the target monocular image of the target field crop, and updating the target monocular image, thereby improving the accuracy of the target monocular image input to the three-dimensional reconstruction model. Further, the present scheme further removes background points in the target three-dimensional color point cloud by performing foreground segmentation on the target three-dimensional color point cloud after obtaining the target three-dimensional color point cloud of the target field crop, and updating the target three-dimensional color point cloud, thereby improving the accuracy of three-dimensional reconstruction of the field crop.

[0064] In an optional embodiment of the present application, the target monocular image includes a target reference object. Correspondingly, after obtaining the target three-dimensional color point cloud of the target field crop, the method further includes: performing size and orientation calibration on the target three-dimensional color point cloud according to the size and orientation of the target reference object, and updating the target three-dimensional color point cloud.

[0065] The target reference object is used to calibrate the size and orientation of the target three-dimensional color point cloud, so that the target three-dimensional color point cloud is consistent with the size and orientation of the target field crop.

[0066] Specifically, based on the target reference object, the actual size of the target field crop is determined, and then the target three-dimensional color point cloud is converted to the actual size. Based on the orientation of the target reference object, the target three-dimensional color point cloud is rotated to an orientation consistent with the ground coordinate system, and the target three-dimensional color point cloud is updated.

[0067] Optionally, based on the target three-dimensional color point cloud, a phenotypic shape parameter of the target field crop is calculated. The phenotypic shape parameter of the target field crop may include plant height, leaf area, stem thickness, etc.

[0068] Wherein, the specific steps of plant height calculation include: projecting the target three-dimensional color point cloud to the vertical axis (Z axis), finding the height difference between the highest point and the lowest point, that is, the height of the plant. If the ground plane has been calibrated to the Z=0 plane, the highest point Z value is the plant height directly. For high-precision applications, multiple sampling or interpolation can be used to improve measurement stability.

[0069] Wherein, the specific steps of leaf area calculation include: first identifying the point set in the target three-dimensional color point cloud that belongs to the leaf. The algorithm based on the normal orientation and position height of the points can be used to determine the points near the horizontal and a certain distance from the stem as the leaf surface. Then, two-dimensional projection or triangular mesh reconstruction is performed on each leaf point set to calculate its surface area. The total leaf area of the plant is obtained by adding the areas of each leaf. If necessary, the sum of the upper and lower surface areas of the leaf (for unfolded leaves, the sum of the two areas is about twice the projected area, depending on the degree of leaf curvature) can be considered.

[0070] Wherein, the specific steps of stem thickness calculation include: in the target three-dimensional color, extracting the point cloud of the stem part (usually a cluster of high-density points located in the center of the plant and extending along the height). At different heights, several cross sections are taken, and the points on the cross section are fitted into a circle or an ellipse, and the diameter / short axis length of the circle or ellipse represents the stem thickness at that height. Generally, the diameter of the cross section near the base (a certain height from the ground, for example, 5 cm) is selected as the stem thickness indicator. If the stem curvature is also needed, it can be obtained by analyzing the trajectory of the stem center line in the point cloud.

[0071] Wherein, the specific steps of other phenotypic trait parameters include: for different target field crop research needs, other phenotypic trait parameters include: branch / tiller number (statistical by identifying the number of stem bifurcation points or base sprouting tillers), crown angle (calculate the angle between the stem and the vertical direction to evaluate the straightness or lodging degree of the plant type), and biomass estimation (approximate the plant biomass by the total volume or coverage area of the target three-dimensional color point cloud).

[0072] The present scheme sets the target single-view image to include a target reference object, and based on the size and orientation of the target reference object, the size and orientation of the target three-dimensional color point cloud are calibrated, improving the accuracy of subsequent crop phenotype analysis.

[0073] Embodiment Two

[0074] Figure 2A flowchart of a field crop three-dimensional reconstruction method provided for the second embodiment of the present application. The second embodiment of the present application is based on the above-mentioned embodiments, and introduces a general data set for pre-training an initial deep learning model, and obtains a corresponding three-dimensional reconstruction model, thereby improving the accuracy of the three-dimensional reconstruction model, and further improving the efficiency and accuracy of the field crop three-dimensional reconstruction. It should be noted that the parts not described in detail in the second embodiment of the present application can be referred to the descriptions of other embodiments.

[0075] Referring to Figure 2 The field crop three-dimensional reconstruction method shown in the above-mentioned embodiments comprises the following steps.

[0076] S201, obtaining a general data set.

[0077] The general data set is used for pre-training a three-dimensional reconstruction model. For example, the general data set comprises a first sample multi-view image of a sample object in a sample scene and a corresponding first sample three-dimensional color point cloud. Optionally, the sample scale of the general data set reaches hundreds of thousands of samples. The sample object is the object of three-dimensional reconstruction. The categories of the sample object are multiple. For example, the sample object is a daily object, such as a plant model. The sample scene is the scene for pre-training the three-dimensional reconstruction model. The categories of the sample scene are multiple. For example, the sample scene comprises an indoor scene and an outdoor scene. The outdoor scene comprises a road scene, a park scene or a field scene, etc. The first sample multi-view image is a multi-view image of the sample object. The first sample three-dimensional color point cloud is a three-dimensional color point cloud for pre-training the three-dimensional reconstruction model, which is constructed based on the first sample multi-view image.

[0078] Specifically, the general data set comprising the first sample multi-view image of the sample object in the sample scene and the corresponding first sample three-dimensional color point cloud is obtained.

[0079] S202, splitting the first sample multi-view image to obtain at least two first sample single-view images of the sample object in the sample scene.

[0080] The first sample multi-view image comprises at least two first sample single-view images. The first sample single-view images are different in view angle. The first sample single-view image is a single-view image of a certain view angle of the sample object.

[0081] Specifically, the first sample multi-view image is split according to different view angles to obtain at least two first sample single-view images corresponding to the sample object in the sample scene.

[0082] S203, calculating the first sample space coordinates and the first sample observation directions of the sample scene according to the first sample three-dimensional color point cloud.

[0083] The first sample space coordinates are space coordinates determined based on the first sample three-dimensional color point cloud. Optionally, the first sample space coordinates comprise measured values or calculated values based on the first sample three-dimensional color point cloud. The first sample observation directions are observation directions determined based on the first sample three-dimensional color point cloud. Optionally, the first sample observation directions comprise measured values or calculated values based on the first sample three-dimensional color point cloud.

[0084] Optionally, the first sample space coordinates of the sample scene are directly read based on the first sample three-dimensional color point cloud. The pose of the acquisition device corresponding to the first sample three-dimensional color point cloud is obtained, and the first sample observation directions corresponding to the first sample space coordinates of the first sample three-dimensional color point cloud are calculated based on the pose of the acquisition device.

[0085] Optionally, the first sample color and the first sample body density of the sample scene are implicitly inferred according to the first sample three-dimensional color point cloud. The first sample space coordinates and the first sample observation directions are calculated based on the first sample body density and the first sample color.

[0086] S204, pre-training an initial deep learning model according to the at least two first sample single-view images corresponding to the sample object in the sample scene, the first sample space coordinates and the first sample observation directions of the sample scene, to obtain a three-dimensional reconstruction model.

[0087] The initial deep learning model is a deep learning model that has not been trained. For example, the initial deep learning model is a deep neural network with a Transformer architecture.

[0088] Specifically, the first sample single-view image corresponding to the sample object in the sample scene is input into the initial deep learning model to obtain a first predicted spatial coordinate and a first predicted observation direction predicted by the initial deep learning model. The initial deep learning model is pre-trained according to the difference between the first predicted spatial coordinate and the first sample spatial coordinate and the difference between the first predicted observation direction and the first sample observation direction until the convergence of the difference between the first predicted spatial coordinate and the first sample spatial coordinate and the difference between the first predicted observation direction and the first sample observation direction, and a pre-trained three-dimensional reconstruction model is obtained.

[0089] S205, acquiring a target single-view image of a target field crop.

[0090] S206, using the three-dimensional reconstruction model to perform three-dimensional information reasoning on the target single-view image to obtain a reference spatial coordinate and a reference observation direction of the target field scene.

[0091] S207, determining a reference color and a reference bulk density of the target field scene according to the reference spatial coordinate and the reference observation direction of the target field scene.

[0092] S208, performing three-dimensional reconstruction on the target field crop according to the reference color and the reference bulk density of the target field scene to obtain a target three-dimensional color point cloud of the target field crop.

[0093] The target three-dimensional color point cloud includes a target spatial coordinate and a target color.

[0094] The technical scheme of the embodiment of the application introduces a general data set to pre-train the initial deep learning model, obtains at least two first sample single-view images corresponding to the sample object in the sample scene by splitting the first sample multi-view image, calculates the first sample spatial coordinate and the first sample observation direction of the sample scene according to the first sample three-dimensional color point cloud, and pre-trains the initial deep learning model according to the at least two first sample single-view images corresponding to the sample object in the sample scene, the first sample spatial coordinate and the first sample observation direction of the sample scene, to obtain a corresponding three-dimensional reconstruction model, thereby improving the accuracy of the three-dimensional reconstruction model and further improving the efficiency and accuracy of the three-dimensional reconstruction of the field crop.

[0095] In an optional embodiment of the present application, after obtaining the three-dimensional reconstruction model, the method further comprises: obtaining a crop-specific dataset; wherein the crop-specific dataset comprises a second sample multi-view image of a sample crop in a sample field scene and a corresponding second sample three-dimensional color point cloud; splitting the second sample multi-view image to obtain at least two second sample single-view images corresponding to the sample crop in the sample field scene; using the three-dimensional reconstruction model to perform three-dimensional information reasoning on the second sample single-view images to obtain a second sample spatial coordinate and a second sample observation direction of the sample field scene; determining a second sample color and a second sample bulk density of the sample field scene according to the second sample spatial coordinate and the second sample observation direction of the sample field scene; performing three-dimensional reconstruction on the sample crop according to the second sample color and the second sample bulk density of the sample field scene to obtain a second predicted three-dimensional color point cloud of the sample crop; and fine-tuning the three-dimensional reconstruction model according to the difference between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud.

[0096] The crop-specific dataset is used to fine-tune the three-dimensional reconstruction model. For example, the crop-specific dataset comprises a second sample multi-view image of a sample crop in a sample field scene and a corresponding second sample three-dimensional color point cloud. The sample crop is the crop used to fine-tune the three-dimensional reconstruction model. The sample crops belong to multiple categories. For example, the sample crops include corn, sorghum, wheat, and soybean. The sample crops correspond to different growth periods. The sample crops have different postures. The sample field scene is the field scene used to fine-tune the three-dimensional reconstruction model. The three-dimensional reconstruction model is fine-tuned by the crop-specific dataset, so that the three-dimensional reconstruction model performs better in the crop field. The second sample multi-view image is a multi-view image of the sample crop. The second sample three-dimensional color point cloud is a three-dimensional color point cloud used to fine-tune the three-dimensional reconstruction model, which is constructed based on the second sample multi-view image. The second sample multi-view image comprises at least two second sample single-view images. The second sample single-view images have different viewing angles. Each second sample single-view image is a single-view image of the sample crop at a certain viewing angle.

[0097] The second sample space coordinate is an inference result of a space coordinate of the second sample single-view image. The second sample space coordinate is a three-dimensional coordinate of a certain space point in the sample field scene. The second sample observation direction is an inference result of an observation direction of the second sample single-view image. The second sample observation direction is an observation direction of a certain space point in the sample field scene. The second sample color is a color intensity used to represent light emitted or scattered along the second sample observation direction at the space point corresponding to the second sample space coordinate in the sample field scene. The second sample color simulates the visual effect of the sample field crop surface. The second sample color is used to render the sample field crop. The second sample volume density is used to represent the absorption or scattering ability of the space point corresponding to the second sample space coordinate to light in the sample field scene. The second sample volume density is used to represent the probability density of the light being intercepted by the medium particles at the space point corresponding to the second sample space coordinate in the sample field scene. The second sample volume density is used to extract the three-dimensional structure of the sample field crop. The second predicted three-dimensional color point cloud is used to represent the model prediction result of the corresponding three-dimensional color point cloud of the sample field crop in the sample field scene.

[0098] Specifically, a crop-specific dataset containing a second sample multi-view image of the sample field crop in the sample field scene and a corresponding second sample three-dimensional color point cloud is obtained. The second sample multi-view image is split according to different views to obtain at least two second sample single-view images corresponding to the sample field crop in the sample field scene. The second sample single-view image is input into the three-dimensional reconstruction model to output the second sample space coordinate and the second sample observation direction of the sample field scene. The second sample color and the second sample volume density of the sample field scene are inferred based on the second sample space coordinate and the second sample observation direction of the sample field scene using a neural radiance field model. The pixel color of a single second predicted space coordinate is determined according to the second sample color and the second sample color of the single second predicted space coordinate using a volume rendering formula. The second predicted color of each second predicted space coordinate is determined according to the pixel color of each second predicted space coordinate. The second predicted three-dimensional color point cloud of the sample field crop is determined according to the second predicted color of each second predicted space coordinate. The difference between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud is calculated, and the three-dimensional reconstruction model is fine-tuned until the difference between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud converges, thereby obtaining the fine-tuned three-dimensional reconstruction model.

[0099] In an optional embodiment of the present application, the three-dimensional reconstruction model is fine-tuned according to the difference between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud, including: determining a reconstruction error according to the difference between the second predicted spatial coordinates of the second predicted three-dimensional color point cloud and the second actual spatial coordinates of the second sample three-dimensional color point cloud; determining a rendering error according to the difference between the second predicted color of the second predicted three-dimensional color point cloud and the second actual color of the second sample three-dimensional color point cloud; and synthesizing the reconstruction error and the rendering error to obtain a synthesis error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud; and fine-tuning the three-dimensional reconstruction model according to the synthesis error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud.

[0100] The second predicted three-dimensional color point cloud includes second predicted spatial coordinates and a second predicted color. The second predicted spatial coordinates are a prediction result of the spatial coordinates of the sample field crop itself in the sample field scene. The second actual spatial coordinates are spatial coordinates determined based on the second sample three-dimensional color point cloud. Optionally, the second actual spatial coordinates include measured values or calculated values based on the second sample three-dimensional color point cloud. The reconstruction error is used to represent the difference between the second predicted spatial coordinates and the second actual spatial coordinates. The second predicted color is a prediction result of the color corresponding to the second predicted spatial coordinates. The second predicted color is used to represent the color intensity of the light emitted or scattered at the spatial point corresponding to the second sample spatial coordinates along the second sample observation direction in the sample field scene. The second predicted color is used to simulate the visual effect of the surface of the second predicted field crop. The second actual color is a rendering color of the second actual spatial coordinates in the second sample three-dimensional color point cloud. The rendering error is used to represent the difference between the second predicted color and the second actual color. The synthesis error is a synthesis result of the reconstruction error and the rendering error, for example, a weighted sum result of the reconstruction error and the rendering error.

[0101] Specifically, the Euclidean distance between the second predicted spatial coordinates of the second predicted three-dimensional color point cloud and the second actual spatial coordinates of the second sample three-dimensional color point cloud is calculated to obtain the reconstruction error. The pixel difference between the second predicted color of the second predicted three-dimensional color point cloud and the second actual color of the second sample three-dimensional color point cloud is calculated to obtain the rendering error. The reconstruction error and the rendering error are weighted and summed to obtain the synthesis error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud. The three-dimensional reconstruction model is fine-tuned with the convergence of the synthesis error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud as the target.

[0102] By introducing the reconstruction error between the second predicted spatial coordinates of the second predicted three-dimensional color point cloud and the second actual spatial coordinates of the second sample three-dimensional color point cloud, and the rendering error between the second predicted color of the second predicted three-dimensional color point cloud and the second actual color of the second sample three-dimensional color point cloud, the comprehensive error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud is determined, and the three-dimensional reconstruction model is fine-tuned based on the comprehensive error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud, thereby improving the reconstruction accuracy of the three-dimensional reconstruction model.

[0103] The scheme introduces a crop special data set, splits the second sample multi-view image to obtain at least two second sample single-view images corresponding to the sample field crop in the sample field scene, uses a three-dimensional reconstruction model to perform three-dimensional information reasoning on the second sample single-view image to obtain second sample spatial coordinates and a second sample observation direction of the sample field scene, determines a second sample color and a second sample bulk density of the sample field scene according to the second sample spatial coordinates and the second sample observation direction of the sample field scene, performs three-dimensional reconstruction on the sample field crop according to the second sample color and the second sample bulk density of the sample field scene to obtain a second predicted three-dimensional color point cloud of the sample field crop, and fine-tunes the three-dimensional reconstruction model according to the difference between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud, thereby realizing fine-tuning of the three-dimensional reconstruction model on the basis of the pre-trained three-dimensional reconstruction model, making the three-dimensional reconstruction model more suitable for the three-dimensional reconstruction scene of the crop in the field environment, and improving the accuracy of three-dimensional reconstruction of the crop in the field environment.

[0104] Embodiment Three

[0105] Figure 3 A flowchart of a three-dimensional reconstruction method of a field crop is provided for the third embodiment of the present application. The third embodiment of the present application is based on the above-mentioned embodiments, Figure 3 A preferred embodiment is provided. It should be noted that parts not described in detail in the third embodiment of the present application can be referred to the descriptions of other embodiments.

[0106] Referring to Figure 3 The three-dimensional reconstruction method of a field crop is shown in the figure, which comprises:

[0107] S1, image data acquisition.

[0108] The target single-view image is a photo of a single field crop taken at close range, for example, an RGB image of a single field crop, to ensure that the target field crop is clearly presented in the target single-view image.

[0109] Specifically, the target single-view image of the target field crop is acquired from the field environment by an image acquisition module. For example, a variety of image acquisition devices are used, such as a mobile terminal such as a smart phone or a tablet computer, a camera to take a photo of a single plant, or a fixedly installed monitoring camera to capture an image of a single field crop at a time.

[0110] Optionally, the relevant information of the shooting is also recorded for later use. For example, basic camera parameters and metadata, etc. The metadata includes the time, place and distance estimation of the shooting. The camera parameters include focal length and resolution, etc.

[0111] Optionally, for a scene that needs to acquire real scale information, a simple target reference object (for example, a ruler with a known size is placed beside the target field crop and shot together) is introduced so as to calibrate the actual size of the three-dimensional model later.

[0112] S2, image data processing.

[0113] Specifically, the original image of the acquired target single-view image is preprocessed by an image processing module. For example, the preprocessing operation includes but is not limited to size adjustment, color correction and denoising, etc. to obtain an image with optimized quality, thereby improving the three-dimensional reconstruction quality and robustness. The size adjustment is the image resolution adjustment, such as scaling and cropping according to the input requirements of the three-dimensional reconstruction model. The color correction includes color space conversion and normalization, such as normalizing the RGB value to the [0, 1] interval. A filtering algorithm or a dehazing algorithm can be used for denoising to improve the image quality and avoid the problems of uneven light or motion blur that may exist in the field environment.

[0114] Optionally, if there are complex backgrounds or multiple plants in the target single-view image, foreground segmentation can be further performed to separate the target field crop from the background, extract the image region of the target field crop and generate a mask, and update the target single-view image to avoid the interference of background debris on the three-dimensional reconstruction. For example, a classical image segmentation algorithm or a deep learning model can be used to realize foreground segmentation, and the crop foreground mask generated will be input into the three-dimensional reconstruction model to filter the non-crop area during three-dimensional reconstruction. Optionally, if there is no separate foreground segmentation step, the three-dimensional reconstruction model itself also has a certain robustness and can ignore the background influence to a certain extent. After this step, the output is the target single-view image suitable for input into the three-dimensional reconstruction model.

[0115] S3, multi-view image inference.

[0116] The three-dimensional reconstruction model is the core of the entire field crop three-dimensional reconstruction method, and a large-scale reconstruction model based on deep learning infers three-dimensional information from the input target monocular image. The neural network inside the three-dimensional reconstruction model adopts a Transformer architecture, and its self-attention mechanism can combine the overall layout and detailed texture of the target monocular image to infer the spatial structure. Since the three-dimensional reconstruction model has learned rich three-dimensional priors, it can "infer" the reasonable shape of the occluded surface and the unseen angle from a single target monocular image. It should be emphasized that the three-dimensional reconstruction model has strong generalization ability after fine-tuning on crop data, and can give reasonable three-dimensional reconstruction results for crops of different types and different growth stages.

[0117] For example, the pre-training and fine-tuning process of the three-dimensional reconstruction model includes:

[0118] First, a large general-purpose dataset is used to pre-train the three-dimensional reconstruction model. The dataset contains multi-view images and corresponding sample three-dimensional color point clouds of various sample objects and sample scenes, with a scale of hundreds of thousands of samples. The sample objects include daily objects and plant models, etc. The sample scenes include common indoor and outdoor scenes. The three-dimensional reconstruction model can be a Transformer reconstruction model. The training method is a combination of self-supervised and supervised methods, which enables the model to learn the ability to infer spatial structure from images. The pre-training stage consumes a large amount of GPU (Graphics Processing Unit) computing power, but gives the three-dimensional reconstruction model general three-dimensional perception ability.

[0119] Subsequently, the three-dimensional reconstruction model is fine-tuned using a crop-specific dataset. The crop-specific dataset includes photos of crops such as corn, sorghum, wheat, and soybeans in different growth stages and different poses, as well as high-precision point cloud ground truths obtained through multi-view reconstruction or laser scanning. The pre-trained three-dimensional reconstruction model is fine-tuned using supervised learning with the crop-specific dataset to optimize its performance in the crop domain. In the fine-tuning stage, the loss function is composed of reconstruction error (such as the distance between the predicted point cloud and the true point cloud) and rendering error (the pixel difference between the predicted view synthesis image and the true image). During the training process, we found that the three-dimensional reconstruction model gradually learned the growth structure characteristics of plants, such as the leaf alternation rule of corn, the ear shape contour of sorghum, and the cluster of small spike layout of wheat, etc. After training, the three-dimensional reconstruction model parameters are frozen for inference and deployment. Experiments show that the model fine-tuned on the crop dataset has significantly improved reconstruction accuracy compared to the general model without fine-tuning: for example, on the corn validation set, the average Euclidean distance error between the single-image reconstructed point cloud and the true point cloud is reduced by more than 30%, and the estimation error of plant height and other indicators is reduced from about 5% to within 2%. This proves the importance of the fine-tuning process.

[0120] Specifically, by the three-dimensional reconstruction module, the preprocessed target single-view image is input into a pre-trained and fine-tuned large-scale reconstruction model (LRM), i.e., a three-dimensional reconstruction model, to directly output the NeRF implicit representation corresponding to the target field scene, i.e., the reference space coordinates and the reference observation direction. The reference space coordinates and the reference observation direction are essentially a set of parameters that can be used for volume rendering. The reference space coordinates and the reference observation direction contain the information of the three-dimensional structure and the appearance texture of the target field crop. The reference space coordinates and the reference observation direction are usually defined by a multi-layer perception (MLP) network, but in the present application, the reference space coordinates and the reference observation direction are directly given by the three-dimensional reconstruction model, rather than being obtained by pixel-by-pixel optimization.

[0121] For example, the target single-view image is encoded by several layers of convolution or Transformer network structure of the three-dimensional reconstruction model to extract multi-scale features of the image, and then the multi-scale features are mapped to the NeRF (Neural Radiance Field) implicit representation, i.e., the reference space coordinates and the reference observation direction, by a decoding network. Specifically, when the input image resolution or scene scale is large, the Transformer architecture flexibly adjusts the computational overhead (for example, by block or hierarchical processing), so the three-dimensional reconstruction model has good scalability and is suitable for high-resolution remote sensing images or larger-scale scene reconstruction.

[0122] S4, three-dimensional point cloud generation.

[0123] Specifically, by the point cloud generation module, the reference space coordinates and the reference observation direction output by the three-dimensional reconstruction model are used to generate the target three-dimensional color point cloud of the target field crop by using a volume rendering algorithm.

[0124] For example, a volume rendering algorithm is used to uniformly sample in space, for each sampling point, the reference color and the reference volume density are inferred by using the NeRF (Neural Radiance Field) model. When the reference volume density exceeds a certain threshold, it is determined that the sampling point belongs to the surface of the target field crop. In this way, the target three-dimensional point cloud of the target field crop can be extracted. Each point in the target three-dimensional color point cloud has RGB color information, forming a target three-dimensional color point cloud with texture. The target three-dimensional color point cloud contains the (x, y, z) target space coordinates and the RGB target color of each point.

[0125] S5, point cloud data post-processing.

[0126] Specifically, by the point cloud post-processing module, a series of post-processing operations are performed on the initially generated target three-dimensional color point cloud. For example, the post-processing operations include denoising, hole filling, smoothing, coordinate calibration, and foreground segmentation.

[0127] For example, first, if there is a crop foreground mask, only the crop points within the crop foreground mask range are retained according to the crop foreground mask, and the background points in the target three-dimensional color point cloud are removed, only the target field crop itself is retained. Then, the target three-dimensional color point cloud is denoised to remove obvious noise points (or isolated points) and outliers (or floating debris), and the coherence of the target three-dimensional color point cloud is ensured to improve the cleanliness of the target three-dimensional color point cloud. Second, the target three-dimensional color point cloud is hole filling, and for local missing areas, the holes can be filled by interpolation or neighborhood algorithm. Next, a smoothing filter or surface reconstruction algorithm is applied to adjust the position distribution of the smoothed points, reduce the jitter caused by reconstruction error, reduce the burr in the target three-dimensional color point cloud, and make the surface of the target three-dimensional color point cloud smoother. If a continuous surface is needed, the target three-dimensional color point cloud can be further triangulated to obtain a mesh model. Finally, the target three-dimensional color point cloud is calibrated in scale and orientation (i.e. coordinate calibration), and the point cloud coordinate system is converted to real scale and unified direction, for example, the Z axis points to the vertical direction, and the unit is millimeter. For example, the ratio of pixels to actual length is calculated by the target reference, the point cloud coordinates are converted to real scale units, and the target three-dimensional color point cloud is rotated / translated to make it conform to the ground coordinate system (such as bottom alignment with the ground plane).

[0128] Optionally, for structures that need further analysis, component separation can also be performed, such as distinguishing stems from leaves in the target three-dimensional color point cloud based on height and density, to prepare for subsequent phenotype index calculation of the target field crop.

[0129] S6, phenotype parameter extraction.

[0130] Among them, the phenotype parameters (i.e. phenotype shape parameters) include plant height, leaf area, and stem thickness, etc.

[0131] Specifically, the processed target three-dimensional color point cloud is analyzed by the phenotype index extraction module to calculate various phenotype trait parameters of the target field crop. The device automatically reads the point cloud data of the target three-dimensional color point cloud and calls the corresponding algorithm module to complete the calculation. The results are saved in numerical form and descriptive statistics (such as average leaf area or maximum leaf length, etc.) are formed. In this process, the device may need to further split the device (such as distinguishing leaf and stem points) to assist manual index calculation.

[0132] For example, the specific steps of plant height calculation include: projecting the target three-dimensional color point cloud onto the vertical axis (Z axis) to find the height difference between the highest point and the lowest point, which is the height of the plant. If the ground plane has been calibrated to the Z=0 plane, the highest point Z value is directly the plant height. For high-precision applications, multiple sampling or interpolation can be used to improve measurement stability.

[0133] The specific steps of leaf area calculation include: first, identify the point set belonging to leaf in the target 3D color point cloud. Points near the horizontal and a certain distance from the stem can be determined as the leaf surface by using the algorithm based on the normal orientation and position height of the points. Then, project or reconstruct the triangular mesh for each leaf point set to calculate its surface area. Add up the area of each leaf to get the total leaf area of the plant. If necessary, the sum of the upper and lower surfaces of the leaf (the sum of the two areas is about twice the projected area, depending on the degree of leaf curvature) can be considered.

[0134] The specific steps of stem diameter calculation include: in the target 3D color, extract the point cloud of the stem part (usually a cluster of high-density points located in the center of the plant and extending along the height). Take several cross sections at different heights, and fit the points on the cross section as a circle or an ellipse, whose diameter / short axis length represents the stem diameter at that height. Generally, the diameter of the cross section near the base (a certain height from the ground, for example, 5 cm) is selected as the stem diameter indicator. If needed, the stem curvature, etc. can also be obtained by analyzing the trajectory of the stem center line in the point cloud.

[0135] The specific steps of other phenotypic trait parameters include: according to the research needs of different target field crops, other phenotypic trait parameters include: branch / tiller number (statistical by identifying the number of stem bifurcation points or base of tillers), canopy angle (calculate the angle between the stem and the vertical direction to evaluate the straightness or lodging degree of the plant type), and biomass estimation (approximate the plant biomass by the total volume or coverage area of the target 3D color point cloud), etc.

[0136] S7, result output and storage.

[0137] Specifically, the 3D reconstruction results of the target field crop and the analyzed phenotypic trait parameters are output through the data storage module.

[0138] On the one hand, the final target 3D color point cloud and the original target monocular image are stored in a designated location to support future review, visualization or further processing (such as using the point cloud for digital breeding simulation), and the phenotypic trait parameters and related metadata (such as measurement time, location and crop variety, etc.) are saved in a database record or file associated with the identity information of the target field crop. Support recording data by batch or test, and provide retrieval interface to make it convenient for users to query historical measurement results or export data for statistical analysis. Considering the field application environment, the designated location includes local disk or cloud database. The local disk can be used for local storage. For example, the local disk includes device hard disk and SD card, etc. The cloud database can be a network database. If deployed on a cloud server, the cloud database and data management platform can be further utilized to realize multi-user collaboration and large-scale data integration.

[0139] On the other hand, the three-dimensional reconstruction result of the target field crop is presented to the user by the visualization interaction module through a user interface, including displaying the target three-dimensional color point cloud in a three-dimensional window and listing the phenotypic trait parameters on a panel. The user can interactively view the target three-dimensional color point cloud, view the morphological characteristics of the target field crop from different angles, and also export the point cloud or report. The output phenotypic trait parameters are usually given in numerical form, with units (such as plant height in centimeters, leaf area in square centimeters, stem diameter in millimeters, etc.). If uncertainty needs to be measured, the confidence interval or error estimate of the phenotypic trait parameter calculation can also be provided (for example, estimate the error range of ± according to the noise level of the point cloud).

[0140] Through an intuitive user interface, the user performs the following operations: uploading / capturing images, starting reconstruction analysis, viewing three-dimensional models and index results, and managing data, etc. In terms of result display, the point cloud or reconstruction grid is drawn using three-dimensional rendering technology, which can present a realistic target three-dimensional color point cloud of the target field crop on the screen, and the user can rotate the viewing angle, zoom in and out to observe the details. At the same time, key phenotypic trait parameters such as "plant height: 180 cm, leaf area: 4300 cm², stem diameter: 2.1 cm" are displayed on the side panel or next to the model, so that the user can easily obtain numerical information. If some parameters are outside the expected range, the interface can use highlights or warning markers to prompt. In addition, simple data analysis tools can also be provided, such as column chart or line chart comparison of measurement results of multiple crops, to help breeders find differences between different strains. Overall, the visualization interaction module greatly improves the usability of the system, so that even non-professionals can complete the operation and understand the results with the help of the user interface.

[0141] The above modules constitute the main functional units of the system of the present application. In actual deployment, these modules are integrated in the same application program in a software manner, or distributed deployment (for example, the image acquisition module is in the mobile terminal APP, the three-dimensional reconstruction module and the point cloud generation module are in the server side, and the three-dimensional reconstruction result is displayed through the webpage or APP). The modules communicate with each other through well-defined data interfaces to ensure smooth connection of the entire system process.

[0142] Optionally, the system of the present application can be deployed locally or in the cloud according to user needs:

[0143] Among them, the local deployment is to integrate the software of all modules on the user's local computer. The typical configuration is a high-performance notebook computer equipped with at least a 4-core CPU, 16GB of memory and a medium-grade GPU (graphics processor). The user can use the system independently in the environment without network by installing the provided application. This way is suitable for breeding experts to obtain data on the spot in the field or greenhouse. For example, in the field test station, technicians can carry a notebook to collect crop photos and run the system on the spot to obtain measurement results. Since there is no need to upload data, the data security and privacy are guaranteed. The software of the local version is optimized for GPU (graphics processor), and the single reconstruction processing time is about 2 seconds, which fully meets the response requirements of on-site use.

[0144] The cloud deployment is to deploy the three-dimensional reconstruction module and the point cloud generation module on a remote server or a cloud computing platform. The user uploads the collected images to the server through the client on the mobile device or the browser, and the server performs reconstruction and analysis calculation, and then returns the results to the user. This cloud service mode is convenient for centralized processing of massive data, and can fully utilize the powerful computing power and storage of the cloud. For example, a breeding company can upload thousands of field photos to the cloud every day, and the server cluster processes in parallel to return all analysis results in a short time, greatly improving work efficiency. Cloud deployment also facilitates software updates and maintenance, and a professional team ensures that the model and algorithm are always in the best state. Considering the need for network transmission, the system will encrypt and compress the uploaded images to ensure data security and transmission efficiency. For unstable network environment in the field, the system also supports edge computing mode, that is, a small server (such as a GPU-equipped industrial computer) is configured in the field, which can realize centralized computing similar to the cloud and is free from dependence on external networks.

[0145] The above deployment methods complete image uploading through a friendly interface and obtain three-dimensional color point cloud and phenotype trait parameters within a few seconds. The present application is carefully designed in model training and system engineering to ensure that it has both advanced algorithm performance and practical operability, and can play a role in real agricultural production environment.

[0146] S8, repeat or batch processing.

[0147] If it is necessary to analyze multiple target field crops in batch, the above steps can be executed in batch mode. The system supports importing multiple target field images at a time, sequentially completing reconstruction and analysis, and displaying all results. For example, in a breeding test, the user takes and imports hundreds of plant photos at a time, and the system will automatically process them in batch and finally output a table of trait data of each plant, greatly improving work efficiency.

[0148] The above steps constitute the complete process of the method of the present application. From the single target monocular image of the target field crop to the target three-dimensional color point cloud to the phenotype data, each stage is supported by a corresponding processing algorithm, and the data transmission and conversion in the process are automatically completed by the system without human intervention.

[0149] In practical applications, some steps are omitted or simplified according to needs, for example, when the target is only to obtain the target three-dimensional color point cloud without the need to calculate specific phenotype indicators, step S6 can not be performed; or when the image quality requirement is not high, the pretreatment of steps S2 and / or S5 can be simplified. However, in general, complete execution of the above process can ensure that high-quality reconstruction results and comprehensive phenotype analysis are obtained.

[0150] The entire technical solution can complete the conversion of field crops from two-dimensional images to three-dimensional color point clouds to phenotype data in an efficient and automated manner, providing convenience for agricultural research and production. Specifically:

[0151] (1) Single-view low-cost reconstruction: Compared with traditional three-dimensional reconstruction methods that require multi-view imaging or special equipment, the present application only needs a single-view image taken by a common RGB camera to complete the reconstruction task, greatly reducing the difficulty of data acquisition and equipment cost, making high-throughput crop phenotype acquisition possible. In practical applications, users only need to use a mobile phone or other readily available devices to take pictures, without the need to purchase expensive sensors or build complex photography systems, demonstrating high economic efficiency and convenience.

[0152] (2) High precision driven by large model: The present application innovatively introduces a pre-trained large-scale reconstruction model (LRM) combined with a Transformer architecture, which is adapted to the field scene through supervised fine-tuning. Benefiting from the rich prior knowledge learned by the large model, the system can infer the complete three-dimensional structure from limited visual information. Compared with traditional algorithms or small networks, the large model of the present application significantly improves the reconstruction accuracy and detail fidelity, and can reconstruct the main leaf and stem structure even for complex plants such as corn, ensuring the reliability of subsequent phenotype measurement.

[0153] (3) Achieve efficient reconstruction: Traditional three-dimensional reconstruction often requires iterative optimization for several minutes or even several hours to obtain results, while the present application directly predicts the reference spatial coordinates and reference observation direction by designing an end-to-end network, and the three-dimensional color point cloud can be obtained by one forward calculation. This improvement greatly speeds up the reconstruction process and can complete the model generation of a single plant in seconds. Therefore, the present system can support the near real-time application needs in the field and obtain measurement results in a timely manner, saving valuable time for agronomic decision-making.

[0154] (4) Color point cloud and texture fidelity: The three-dimensional color point cloud output by the system carries real color texture information, which can realistically reproduce the appearance of crops. This is helpful for human eye verification and intuitive analysis. Unlike some scanning techniques that only obtain structural information, the present invention preserves the color characteristics of plants (such as whether the leaves are healthy can be judged from the color), which provides the possibility for further disease and pest detection, etc. At the same time, color texture helps to accurately segment different organs (such as distinguishing green leaves from brown stems), improving the accuracy of phenotype analysis.

[0155] (5) Integrated extraction of multiple phenotype indicators: The system of the present invention completes three-dimensional reconstruction while extracting multiple key phenotype indicators, realizing an integrated analysis process. Traditionally, different traits often need different tools and methods to measure, such as using a ruler to measure plant height, using a scanner to measure leaf area, and using a caliper to measure stem diameter, while the present system can obtain all these indicators simultaneously through a single three-dimensional modeling. Each indicator is derived from the same model and data source, with consistency and comparability. This integrated measurement reduces tedious repetitive work and reduces error propagation, with great advantages in high-throughput phenotype analysis.

[0156] (6) End-to-end automation and ease of use: The present invention seamlessly integrates image processing, model inference, point cloud generation, feature calculation, and result storage and display into an end-to-end automated process. Users do not need to have computer vision or programming background, and can obtain results simply by operating, which embodies good ease of use. The visual interface provided by the system further reduces the threshold for use, enabling agricultural practitioners to easily adopt the technology. At the same time, the end-to-end architecture reduces the trouble of intermediate data transmission and format conversion, improving processing efficiency and stability.

[0157] (7) Flexible deployment and expansion: The system uses modular design and can be flexibly deployed in local or cloud environment according to needs, meeting the needs of different scale applications. The module interface is clearly defined, and it is also convenient for subsequent function expansion or algorithm upgrade. For example, a new deep learning model can be replaced into the reconstruction module, or a new trait calculation module for specific crops can be added. The present invention provides an open and sustainable development framework for crop phenotype analysis platform, with long-term application value.

[0158] (8) Adaptation to complex field environment: Considering the complex and variable field environment (such as changes in light, cluttered background, and large differences between individual plants), the present invention focuses on robustness in algorithms. For example, using Transformer global modeling to reduce the impact of occlusion, using data augmentation and segmentation to improve the adaptability of the model to cluttered background, and using scale calibration to ensure the comparability of results taken at different distances. These measures enable the system to maintain good performance under non-ideal conditions and are truly suitable for field use.

[0159] (9) Innovation fusion frontier technology: The present application organically fuses a plurality of frontier technologies emerging in recent years to form innovative applications in the field of agriculture. For example, the NeRF implicit representation is introduced into crop phenotype acquisition, realizing a new approach to fine three-dimensional reconstruction; the visual Transformer is used for a three-dimensional reconstruction model, breaking through the limitation of local perception of traditional CNN (convolutional neural network) and improving the global reasoning ability; the large model pre-training + fine-tuning paradigm is adopted to fully utilize the mass data prior to improve the effect of small data set tasks. These innovative combinations make the present application realize a leap in theory and practice compared with existing crop three-dimensional reconstruction methods, providing a new idea for the development of smart agriculture technology.

[0160] In summary, the present application has the outstanding advantages of "low cost, high precision, high efficiency, full automation, multi-function", etc., and solves the long-standing problem of restricting three-dimensional phenotype analysis of field crops. Through the application of the present application, the breeding and agricultural production field will be able to obtain massive and accurate crop trait data more quickly, helping to accelerate the breeding process and optimize agronomic decision-making, and having significant economic and scientific research value. In future popularization and application, the present application is expected to become part of the digital agricultural infrastructure, promoting the transformation of traditional agriculture to data-driven intelligent agriculture.

[0161] For example, a corn plant growing in the field is selected as the object. An operator uses an ordinary smart phone to take a target single-view image (image resolution 4032x3024 pixels) of the front of the plant at a distance of about 2 meters from the plant. The image is input into the system of the present application, and the analysis result is obtained according to the following steps:

[0162] S301, image input and preprocessing.

[0163] Specifically, the system receives the uploaded photo of the corn plant. The image processing module first scales the original photo to the size required by the model (for example, scales the long side to 1024 pixels), and adjusts the brightness and contrast to highlight the plant details. Since the photo background contains other weeds and soil, the system enables the foreground segmentation function: a pre-trained deep learning segmentation model is used to segment the corn plant, and the result successfully extracts the corn plant area from the background and generates a foreground mask. The image retaining only the plant foreground part is sent to the next step.

[0164] S302, LRM model inference.

[0165] Specifically, the three-dimensional reconstruction module loads the pre-trained and fine-tuned LRM model (a neural network with a Transformer architecture, with a parameter size of about 100 million). The system inputs the processed corn plant image into the model, and after multiple layers of calculation, the reference spatial coordinates and reference observation direction of the field scene are generated in about 0.5 seconds, which can accurately depict the structure of the corn plant. The model inference stage uses GPU acceleration, which can be completed in real time on a mid-range graphics card.

[0166] S303, point cloud generation and optimization.

[0167] Specifically, the system generates a target three-dimensional color point cloud of the corn plant from the reference spatial coordinates and reference observation direction output by the model. The specific method is to perform volume rendering sampling at a density of 2 points per pixel in the camera perspective space, resulting in about 150,000 three-dimensional points. These points basically cover the surface of the corn plant, including the stem, leaves, and ear, and each point has an RGB color corresponding to the original image (such as green for leaves and yellow for grain in the ear). The point cloud post-processing module optimizes the initial point cloud: first, it removes a small number of noise points belonging to the background sky according to the foreground mask; then it removes scattered isolated point clouds (mainly pseudo points generated by local reflection of leaf tips) using a neighborhood filtering algorithm; and finally, it performs light smoothing on the point cloud to make the point distribution on the leaf surface more uniform. The result is a fine corn plant color point cloud model containing about 120,000 valid points. As shown in FIG. 6, the target three-dimensional color point cloud of the corn plant clearly presents the main organ morphology of the corn plant at different rotation angles: the upright main stem, the long strip-shaped leaves distributed at different heights, and the corn ear at the top. Figure 4

[0168] S304, feature extraction and measurement.

[0169] Specifically, the phenotype index extraction module analyzes the above-mentioned point cloud and calculates the key trait parameters of the corn plant.

[0170] Plant height: The system retrieves the highest and lowest point coordinates of the point cloud. After calculation, the highest point (located at the top of the corn ear) is about 2.34 meters high, and the lowest point (near the ground at the base of the plant) is about 0.05 meters high. The difference between the two is about 2.29 meters, which is recorded as the plant height of the plant (the system converts it to 229 centimeters output). This result is basically consistent with the result measured by the artificial use of a measuring rod (about 230 centimeters), with an error of less than 1 centimeter, verifying the accuracy of the height direction scale of the reconstruction model.

[0171] ​Leaf area: The system clusters the point cloud by normal direction and height, and classifies the points that are likely to belong to different leaves. As a result, it identifies 12 leaves (not including the small leaves that have just emerged and not yet unfolded) on the plant. For each leaf, the system reconstructs the leaf surface using the alpha shape algorithm and calculates the surface area. The system sums up the surface areas of all the leaves and obtains a total leaf area of about 5100 cm² (0.51 m²). Among them, the largest functional leaf has an area of about 630 cm². This total leaf area is consistent with the typical value of a corn plant of similar height, indicating that the system has achieved high accuracy in estimating leaf area.

[0172] Stem thickness: The system extracts the point cloud of a section of stem near the base of the plant at a height of 10 cm, and fits a circular cross-section using the least squares method. The calculated diameter of the cross-section is about 2.6 cm. This value can be regarded as an indicator of the thickness of the corn stem (the system outputs stem thickness = 26 mm). In addition, the system also calculates the change in the cross-sectional diameter of the stem at every 50 cm from the base to the top, and the result shows that the stem slightly thins out, with a top diameter of about 1.5 cm. This continuous stem thickness distribution is very useful for analyzing the mechanical strength and lodging resistance of the plant.

[0173] In addition to the above indicators, the system also records additional information such as the number of leaves (12) and the maximum leaf length (the length of the main vein of the leaf is about 80 cm) of the plant. All measurements are automatically completed by the algorithm, and the results have been repeatedly verified, with high reliability.

[0174] S305, result output.

[0175] Specifically, the system saves the reconstructed three-dimensional point cloud of the corn plant and the extracted phenotype parameters to the database. The point cloud data is saved in PLY format, with the naming containing the sample number and timestamp; the phenotype data is saved in the system's crop measurement database, including fields such as sample ID, date, plant height (cm), total leaf area (cm²), stem thickness (mm), number of leaves, etc. Subsequently, the system interface displays the results of this analysis to the user: the target three-dimensional color point cloud is rendered in the main window, and the user can drag and rotate the model to view the structure of the corn plant from various angles. The interface on the side lists the measurement values in detail, such as "plant height: 229 cm, total leaf area: 5100 cm², base stem thickness: 26 mm, number of leaves: 12", etc., and the green text label indicates that the plant is growing well and the indicators are within the normal range.

[0176] This embodiment fully demonstrates the effectiveness of the system in actual field conditions. Through a mobile phone photo, the system successfully restores the three-dimensional structure of the corn plant and accurately measures multiple key traits, with efficiency and convenience far exceeding traditional methods.

[0177] For example, the following case describes the specific application of three-dimensional reconstruction of field crops.

[0178] Case 1: High-throughput phenotyping in breeding trials.

[0179] In the process of crop breeding, researchers need to screen excellent materials from numerous varieties and trial combinations. This often requires measuring the agronomic traits of a large number of plants, for example, a corn breeding test field may contain thousands of corns, each of which needs to record plant height, ear height, stem diameter, and leaf area. Under the traditional method, these measurements mainly rely on manual completion, which consumes a lot of manpower and has limited accuracy. After introducing the system of the present application, breeders can greatly improve the measurement efficiency: just hold a mobile phone or camera to quickly take a photo of each corn in the test field (even can be automatically taken by a drone or agricultural machinery), and then batch import the photo into the system. The system will automatically complete the three-dimensional reconstruction and trait calculation of each plant. For example, for a trial of 1000 corns, it may take several days to measure each plant by multiple people in the past. Now, using the system, a computer can process all images and output a complete data table within a few hours. According to this, breeding experts quickly compare the growth performance of different strains, and select excellent single plants with appropriate plant height, strong stem and large leaf area for next generation breeding. Since the measurement data provided by the system is more objective and consistent, it reduces human error and improves the reliability of selection decisions. In addition, the three-dimensional model of each plant saved by the system can also be used for careful observation of plant morphology and comprehensive evaluation with professional knowledge. This application case shows the great potential of the present application in the high-throughput phenotyping platform, which can greatly accelerate the breeding process.

[0180] Case 2: Field crop growth monitoring and digital agricultural decision-making.

[0181] In agricultural production, monitoring the growth status of crops is essential for timely field management. For example, farmers and agricultural technicians need to know the growth of crops at different growth stages, such as height growth, leaf area index change, plant health, etc., to determine whether they need to fertilize, irrigate, control pests and diseases, or adjust planting density, etc. The system of the present invention can be applied to build a digital agricultural monitoring platform: set up point cameras in the field plot or take pictures of crops regularly by field patrol personnel, use the system to generate corresponding three-dimensional models and parameters, and continuously track the growth of crops. Taking a rice field as an example, the technician uses a drone to take close-up photos of part of the rice plants at a fixed height every week, and the system automatically calculates the curves of plant height and tiller number over time. If it is found that the growth of rice plants in a certain block is significantly lagging behind (such as the plant height is much lower than the average height), it may indicate that the soil nutrients are insufficient or diseases have occurred, and the system will issue a warning. The manager can take timely measures such as supplementing fertilizer or spraying pesticides accordingly, so as to avoid yield loss. In drought stress monitoring, the system can reflect the water shortage condition through the degree of leaf curling and the reduction of leaf area, and guide irrigation scheduling. Compared with traditional visual observation or single-point measurement relying on experience, the present system provides a quantitative and visual growth monitoring method, helping farmers to make fine management decisions in a data-driven way. This digital agricultural decision support system turns farmland into a "smart farmland", improving resource utilization efficiency and crop yield.

[0182] Case three: crop phenotype research and variety trait analysis.

[0183] In agricultural research institutions, scientists often need to conduct in-depth research on certain morphological characteristics of crops, such as studying the effects of dense planting on corn plant morphology, analyzing differences in wheat plant type under different irrigation schemes, or exploring how specific gene mutations change plant morphology. The system of the present application can serve as a powerful research tool, providing accurate data support for these studies. Researchers can easily obtain a large number of three-dimensional models of plants using the system and extract detailed morphological parameters from them. For example, a study on the angle of corn leaves (the angle between the leaf and the stem) used to require manual measurement of multiple leaf positions using a protractor, but now the spatial inclination distribution of each leaf can be automatically calculated from the reconstructed three-dimensional model. For another example, the system can also directly obtain the crown structure (such as leaf area distribution, leaf inclination distribution) indicators that are concerned in botanical research through point cloud analysis. Researchers can also use the three-dimensional models output by the system for visual demonstration, such as displaying plant models under different treatments side by side to compare their morphological differences. Furthermore, the data output format of the system facilitates integration with other analysis software, and researchers can import the batch output phenotype data into statistical analysis tools or machine learning models to conduct group-level trait association analysis or growth modeling. In a typical case, a research team used the system to measure the three-dimensional phenotype data of hundreds of rice varieties, combined with genotyping data for genome-wide association study (GWAS), successfully locating key gene loci affecting plant height and tiller number. This demonstrates the assistance of the present application to cutting-edge research, making the acquisition and application of crop phenotype big data more efficient and convenient than ever before.

[0184] As seen from the above typical cases, the field crop three-dimensional reconstruction and phenotype analysis system of the present application has important value in breeding, production, and research, and has opened up a path from image acquisition to digital trait analysis, providing a new intelligent and data-based tool for modern agriculture, which will promote the innovation of agricultural production and research methods.

[0185] Embodiment Four

[0186] Figure 5 A structural schematic diagram of a field crop three-dimensional reconstruction device provided for Embodiment Four of the present application. The embodiment of the present application can be applicable to the case of three-dimensional reconstruction of crops in the field, and the device executes the field crop three-dimensional reconstruction method. The device is realized in the form of hardware and / or software, and can be configured in an electronic device that carries out the function of three-dimensional reconstruction of field crops.

[0187] Reference Figure 5The field crop three-dimensional reconstruction device shown comprises: a target single-view image acquisition module 501, a three-dimensional reconstruction model inference module 502, a reference color calculation module 503, and a target three-dimensional color point cloud reconstruction module 504. The target single-view image acquisition module 501 is configured to acquire a target single-view image of a target field crop; the three-dimensional reconstruction model inference module 502 is configured to perform three-dimensional information inference on the target single-view image by using a three-dimensional reconstruction model to obtain a reference spatial coordinate and a reference observation direction of a target field scene; the reference color calculation module 503 is configured to determine a reference color and a reference volume density of the target field scene according to the reference spatial coordinate and the reference observation direction of the target field scene; and the target three-dimensional color point cloud reconstruction module 504 is configured to perform three-dimensional reconstruction on the target field crop according to the reference color and the reference volume density of the target field scene to obtain a target three-dimensional color point cloud of the target field crop; wherein the target three-dimensional color point cloud comprises a target spatial coordinate and a target color.

[0188] The technical scheme of the embodiment of the present application performs three-dimensional information inference on the target single-view image by using a three-dimensional reconstruction model, breaks through the limitation of local perception of the prior art, improves the global inference capability based on the target single-view image, and only needs a target single-view image of a target field crop to complete the three-dimensional reconstruction of the target field crop. Compared with the traditional multi-view three-dimensional reconstruction and the three-dimensional reconstruction method using a special device, the data acquisition difficulty and the equipment cost are greatly reduced. In actual application, there is no need to add expensive sensors or build a complex image acquisition system, which has high economic efficiency and convenience. At the same time, the single-view image processing, three-dimensional information model inference, and three-dimensional color point cloud generation are seamlessly integrated to form an end-to-end automated process. The three-dimensional reconstruction user does not need to have a computer vision or programming background and only needs to perform simple operations to obtain the result, which has good usability. Moreover, the end-to-end architecture also reduces the errors of intermediate data transmission and conversion, improves the processing efficiency and stability of the field crop three-dimensional reconstruction. Moreover, unlike only acquiring the spatial structure of the target field crop, the technical scheme of the embodiment of the present application outputs the three-dimensional color point cloud of the target field crop to realistically reproduce the appearance of the target field crop, which is convenient for the human eye to verify and subsequent intuitive analysis. Specifically, by retaining the color features of the target field crop, further pest detection is facilitated. At the same time, the color texture helps to accurately segment different organs of the target field crop, improves the accuracy of crop phenotype analysis, and thus improves the applicability in the field of crops.

[0189] In an optional embodiment of the present application, the device further comprises: a general dataset obtaining module, configured to obtain a general dataset before obtaining the single-view image of the target field crop; wherein the general dataset comprises a first sample multi-view image of a sample object in a sample scene and a corresponding first sample three-dimensional color point cloud; a first sample multi-view image splitting module, configured to split the first sample multi-view image to obtain at least two first sample single-view images of the sample object in the sample scene; a first sample spatial coordinate calculation module, configured to calculate a first sample spatial coordinate and a first sample observation direction of the sample scene according to the first sample three-dimensional color point cloud; and a three-dimensional reconstruction model pre-training module, configured to pre-train an initial deep learning model according to the at least two first sample single-view images of the sample object in the sample scene, the first sample spatial coordinate and the first sample observation direction of the sample scene, to obtain the three-dimensional reconstruction model.

[0190] In an optional embodiment of the present application, the device further comprises: a crop-specific dataset obtaining module, configured to obtain a crop-specific dataset after obtaining the three-dimensional reconstruction model; wherein the crop-specific dataset comprises a second sample multi-view image of a sample field crop in a sample field scene and a corresponding second sample three-dimensional color point cloud; a second sample multi-view image splitting module, configured to split the second sample multi-view image to obtain at least two second sample single-view images of the sample field crop in the sample field scene; a second sample spatial coordinate generation module, configured to perform three-dimensional information inference on the second sample single-view image by using the three-dimensional reconstruction model to obtain a second sample spatial coordinate and a second sample observation direction of the sample field scene; a second sample color generation module, configured to determine a second sample color and a second sample bulk density of the sample field scene according to the second sample spatial coordinate and the second sample observation direction of the sample field scene; and perform three-dimensional reconstruction on the sample field crop according to the second sample color and the second sample bulk density of the sample field scene to obtain a second predicted three-dimensional color point cloud of the sample field crop; and a three-dimensional reconstruction model fine-tuning module, configured to fine-tune the three-dimensional reconstruction model according to the difference between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud.

[0191] In an optional embodiment of the present application, the three-dimensional reconstruction model fine-tuning module comprises: a reconstruction error generation unit configured to determine a reconstruction error according to a difference between the second predicted spatial coordinates of the second predicted three-dimensional color point cloud and the second actual spatial coordinates of the second sample three-dimensional color point cloud; a rendering error generation unit configured to determine a rendering error according to a difference between the second predicted color of the second predicted three-dimensional color point cloud and the second actual color of the second sample three-dimensional color point cloud; a comprehensive error generation unit configured to comprehensively combine the reconstruction error and the rendering error to obtain a comprehensive error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud; and a three-dimensional reconstruction model fine-tuning unit configured to fine-tune the three-dimensional reconstruction model according to the comprehensive error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud.

[0192] In an optional embodiment of the present application, the target three-dimensional color point cloud reconstruction module comprises: a body density unit configured to compare the reference body density and the preset body density for a single reference spatial coordinate; a target spatial coordinate determination unit configured to determine the reference spatial coordinate as a target spatial coordinate when the reference body density is greater than the preset body density for the single reference spatial coordinate; a target color determination unit configured to determine a target color corresponding to the target spatial coordinate according to a reference color corresponding to the target spatial coordinate; and a target three-dimensional color point cloud reconstruction unit configured to perform three-dimensional reconstruction on the target field crop according to the target color of each target spatial coordinate to obtain a target three-dimensional color point cloud of the target field crop.

[0193] In an optional embodiment of the present application, the device further comprises: a target monocular image foreground segmentation module configured to perform foreground segmentation on the target monocular image of the target field crop after the target monocular image of the target field crop is obtained, and update the target monocular image; and a target three-dimensional color point cloud foreground segmentation module configured to perform foreground segmentation on the target three-dimensional color point cloud of the target field crop after the target three-dimensional color point cloud of the target field crop is obtained, and update the target three-dimensional color point cloud.

[0194] In an optional embodiment of the present application, the device further comprises: a target three-dimensional color point cloud calibration module configured to perform size and orientation calibration on the target three-dimensional color point cloud of the target field crop according to the size and orientation of the target reference object after the target three-dimensional color point cloud of the target field crop is obtained, and update the target three-dimensional color point cloud; and the target monocular image comprises the target reference object.

[0195] The field crop three-dimensional reconstruction device provided in the embodiments of the present application can perform the field crop three-dimensional reconstruction method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0196] The target monocular image of the target field crop, the general data set and the crop special data set involved in the technical scheme of the embodiment of the present application meet the relevant legal regulations and do not violate public order and good customs.

[0197] Embodiment five

[0198] Figure 6 A structural schematic diagram of an electronic device 600 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0199] As shown in Figure 6 The electronic device 600 includes at least one processor 601 and a memory, such as a read-only memory (ROM) 602, a random access memory (RAM) 603, etc., connected to the at least one processor 601 in communication, where the memory stores a computer program executable by the at least one processor. The processor 601 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 602 or the computer program loaded from the storage unit 608 into the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The processor 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0200] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0201] The processor 601 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the processor 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The processor 601 performs various methods and processes described above, such as the field crop 3D reconstruction method.

[0202] In some embodiments, the field crop 3D reconstruction method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the processor 601, one or more steps of the field crop 3D reconstruction method described above can be performed. Alternatively, in other embodiments, the processor 601 can be configured to perform the field crop 3D reconstruction method by any other suitable means, such as by means of firmware.

[0203] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0204] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a machine or a remote machine or a server.

[0205] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0206] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0207] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0208] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server) service.

[0209] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0210] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for three-dimensional reconstruction of field crops, characterized in that, The method includes: Acquire a single-view image of the target field crop; A three-dimensional reconstruction model is used to perform three-dimensional information reasoning on the single-view image of the target to obtain the reference spatial coordinates and reference observation direction of the target field scene; Based on the reference spatial coordinates and reference observation direction of the target field scene, determine the reference color and reference volume density of the target field scene; Based on the reference color and reference volume density of the target field scene, the target field crop is reconstructed in three dimensions to obtain the target three-dimensional color point cloud of the target field crop; wherein, the target three-dimensional color point cloud includes target spatial coordinates and target color.

2. The method according to claim 1, characterized in that, Before acquiring the single-view image of the target field crop, the method further includes: Obtain a general dataset; wherein, the general dataset includes a first sample multi-view image of a sample object in a sample scene and the corresponding first sample 3D color point cloud; The first sample multi-view image is split to obtain at least two first sample single-view images corresponding to the sample object in the sample scene; Based on the three-dimensional color point cloud of the first sample, calculate the first sample spatial coordinates and the first sample observation direction of the sample scene; Based on at least two first sample single-view images corresponding to the sample object in the sample scene, the first sample spatial coordinates of the sample scene, and the first sample observation direction, the initial deep learning model is pre-trained to obtain a three-dimensional reconstruction model.

3. The method according to claim 2, characterized in that, After obtaining the three-dimensional reconstruction model, the following is also included: Obtain a crop-specific dataset; wherein, the crop-specific dataset includes second-sample multi-view images of sample field crops in a sample field scene and corresponding second-sample three-dimensional color point clouds; The second sample multi-view image is split to obtain at least two second sample single-view images corresponding to the sample field crop in the sample field scene; Using the aforementioned 3D reconstruction model, 3D information reasoning is performed on the second sample single-view image to obtain the second sample spatial coordinates and the second sample observation direction of the sample field scene; Based on the second sample spatial coordinates and the second sample observation direction of the sample field scene, determine the second sample color and the second sample volume density of the sample field scene; Based on the second sample color and second sample volume density of the sample field scene, the sample field crops are reconstructed in three dimensions to obtain the second predicted three-dimensional color point cloud of the sample field crops; The 3D reconstruction model is fine-tuned based on the difference between the second predicted 3D color point cloud and the second sample 3D color point cloud.

4. The method according to claim 3, characterized in that, The step of fine-tuning the 3D reconstruction model based on the difference between the second predicted 3D color point cloud and the second sample 3D color point cloud includes: The reconstruction error is determined based on the difference between the second predicted spatial coordinates of the second predicted 3D color point cloud and the second actual spatial coordinates of the second sample 3D color point cloud. The rendering error is determined based on the difference between the second predicted color of the second predicted 3D color point cloud and the second actual color of the second sample 3D color point cloud; The reconstruction error and the rendering error are combined to obtain the combined error between the second predicted 3D color point cloud and the second sample 3D color point cloud. The three-dimensional reconstruction model is fine-tuned based on the combined error between the second predicted three-dimensional color point cloud and the second sample three-dimensional color point cloud.

5. The method according to claim 1, characterized in that, The step of performing three-dimensional reconstruction of the target field crops based on the reference color and reference volume density of the target field scene to obtain the target three-dimensional color point cloud of the target field crops includes: For a single reference space coordinate, the reference volume density and the preset volume density are compared; For a single reference space coordinate, when the reference volume density is greater than a preset volume density, the reference space coordinate is determined as the target space coordinate. The target color corresponding to the target spatial coordinates is determined based on the reference color corresponding to the target spatial coordinates; Based on the target color of each target spatial coordinate, the target field crop is reconstructed in three dimensions to obtain the target three-dimensional color point cloud of the target field crop.

6. The method according to claim 1, characterized in that, After acquiring the target single-view image of the target field crop, the method further includes: Foreground segmentation is performed on the target single-view image, and the target single-view image is updated; Accordingly, after obtaining the target three-dimensional color point cloud of the target field crop, the method further includes: Perform foreground segmentation on the target 3D color point cloud and update the target 3D color point cloud.

7. The method according to claim 1, characterized in that, The target single-view image includes a target reference object; correspondingly, after obtaining the target three-dimensional color point cloud of the target field crop, the method further includes: Based on the size and orientation of the target reference object, the size and orientation of the target 3D color point cloud are calibrated, and the target 3D color point cloud is updated.

8. A three-dimensional reconstruction device for field crops, characterized in that, The device includes: The target single-view image acquisition module is used to acquire target single-view images of the target field crops; The three-dimensional reconstruction model inference module is used to perform three-dimensional information inference on the single-view image of the target using a three-dimensional reconstruction model, so as to obtain the reference spatial coordinates and reference observation direction of the target field scene; The reference color calculation module is used to determine the reference color and reference volume density of the target field scene based on the reference spatial coordinates and reference observation direction of the target field scene. The target 3D color point cloud reconstruction module is used to perform 3D reconstruction of the target field crop based on the reference color and reference volume density of the target field scene, so as to obtain the target 3D color point cloud of the target field crop; wherein, the target 3D color point cloud includes target spatial coordinates and target color.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the field crop three-dimensional reconstruction method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the field crop three-dimensional reconstruction method according to any one of claims 1-7.

Images