Device and method for processing tree data including image data

The tree data processing device uses a machine learning view synthesis model to address the challenges of self-occlusion and anisotropy in 3D tree model rendering, achieving accurate and noise-reduced 3D views for tree management and trading applications.

WO2024228587A1PCT designated stage expired Publication Date: 2024-11-07ROOTRIX INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/006019
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-05-03
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Existing methods for generating 3D tree models from 2D images using NeRF face challenges in accurately rendering self-occluding objects due to the geometric and anisotropic shape of trees, leading to noise and inaccuracies in 3D rendering.

Method used

A tree data processing device and method that employs a machine learning view synthesis model, including NeRF, illumination visibility MLP, surface normal MLP, BRDF identity MLP, and Albedo MLP, to generate accurate 3D views by masking background, performing volumetric rendering, and optimizing for tree species-specific reflectance and anisotropy, thereby reducing noise and improving rendering accuracy.

Benefits of technology

The solution effectively processes tree data to produce accurate 3D views from 2D images, addressing the issue of self-occlusion and anisotropy, and providing comprehensive 3D data for tree management and trading platforms with improved accuracy and reduced noise.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024006019_07112024_PF_FP_ABST
    Figure KR2024006019_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided is a tree information processing system including a tree data processing device, a tree management system, and a tree transaction platform. The tree data processing device comprises: a tree data processing unit that collects and processes tree data including tree information and multi-view 2D images of a tree and provides processed tree data; a communication unit for providing the processed tree data to the tree management system and the tree transaction platform via a wired or wireless communication network; and a database in which the 3D view data and the tree information are stored as structured data.
Need to check novelty before this filing date? Find Prior Art

Description

Device and method for processing tree data including image data

[0001] The present invention relates to a tree data processing device and method, and more particularly, to a device and method for processing one or more tree data including image data of a tree.

[0002] Providing accurate tree information is essential for tree management or online sales. A fundamental prerequisite for tree management, sales, and evaluation is obtaining and providing accurate tree species, specifications, shapes, and images. However, securing this tree data and building an integrated database has traditionally been difficult.

[0003] Meanwhile, building 3D models of trees and providing them on sales platforms can provide more realistic images, and these 3D models can also be used to extract tree specifications and shapes. LiDAR scan data can be used to build 3D models of trees. However, tree point cloud data from LiDAR scans is not only large in volume, but also requires significant computational resources for data processing. Furthermore, it lacks color luminance information, leading to limitations.

[0004] Photogrammetry can also be used to extract 3D objects from images in the form of voxels, point clouds, or meshes. To automate 3D modeling, feature points are extracted from images, their 3D spatial positions are estimated, and a depth map of the 3D shape is generated based on these points, allowing 3D meshing to be performed. However, this method can suffer from problems such as poorly representing the shape of an object depending on its characteristics.

[0005] Recently, a 3D model construction method that generates a continuous synthetic view from 2D images captured from different directions by NeRF (Neural Radiance Fields for view synthesis, hereinafter referred to as "NeRF") has been proposed. The NeRF technique, as described in the paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis by Mildenhall et al. (arXiv:2003.08934v1), is a technology for generating 3D views from 2D images captured in a controlled environment. NeRF is a lighter model than 3D modeling using lidar data and can also extract features that are difficult to obtain from lidar data. NeRF is a view synthesis method for constructing a 3D image from 2D images, and it can render realistic images from a new perspective by representing a 3D scene using continuous 5D features.

[0006] However, when synthesizing views of trees from 2D images using NeRF, the geometric and anisotropic shapes of the trees can make it difficult to implement self-occlusion objects. Specifically, self-occlusion occurs when leaves obscure branches, and the anisotropic shape of the leaves can cause inaccurate 3D rendering using NeRF and generate noise.

[0007] An object of the present invention is to provide a method and system for effectively processing tree data.

[0008] The present invention also provides a method and device for constructing a machine learning view synthesis model for generating a three-dimensional image from a two-dimensional image of a tree, and providing a 3D view with excellent accuracy by removing noise.

[0009] To solve the above problem, a tree data processing device according to a preferred embodiment of the present invention is provided. The tree data processing device includes a processor and a memory, and is a device for processing collected tree data, and includes a tree data processing unit for processing the tree data and providing processed data; a communication unit for communicating with an external device via a wired or wireless communication network; and a database. The tree data includes a multi-viewpoint two-dimensional image of the tree and tree information, and the tree data processing unit includes a three-dimensional view generation unit.

[0010] The above 3D view generation unit extracts tree image data including an area including a tree by masking or deleting the background from the multi-view 2D image, inputs the tree image data into a machine learning view synthesis model, performs volume rendering on the output values, and outputs 3D view data. The 3D view data is preferably 3D view data. It is preferable that the 3D view data and tree information are stored as structured data in the database. In the horizontal aspect, the 3D view data includes views at all angles of 2πsr (steradian) and also views viewed from above. That is, it includes view data at all angles (hemisphere above the ground surface) except for the bottom of the tree. Therefore, the 3D view data here may include 3D view data at all solid angles except for the bottom of the tree. For example, the 3D view data includes view data in any direction of the upper hemisphere of a solid angle 2π (sr), which is the upper half of the total solid angle 4π (sr).

[0011] The above tree data processing unit further includes at least one of a tree object classification unit, a tree shape analysis unit, and a tree standardization unit, and the tree data may further include tree scan lidar data and tree spectral images.

[0012] The above tree object classification unit extracts each of the collected two-dimensional images of trees, tree scan lidar data, or tree spectral images into individual tree area images.

[0013] The above tree shape analysis unit detects the shape of a tree using at least one of tree scan lidar data, tree shape spectral image, and tree 2D image data.

[0014] The tree standardization unit derives the tree standard from tree data, and the tree standardization unit includes tree height (H), crown width (W), and diameter (R, B). The tree standardization unit extracts the tree standard using individual tree data extracted by the tree object classification unit and a tree standardization model. It is preferable that the tree standardization model is generated by machine learning using at least one of a two-dimensional image of a tree, tree scan lidar data, and tree spectral image.

[0015] The above tree data processing device may further include a tree shape analysis unit that detects the shape of a tree using the three-dimensional view data, and a tree standardization unit that calculates the specifications of the tree using the three-dimensional view data.

[0016] The machine learning view synthesis model may include a general NeRF for estimating volume density at each location from a multi-viewpoint two-dimensional image; a lighting visibility multi-layer perceptron (MLP), a surface normal MLP, a BRDF identity MLP, and an Albedo MLP, each of which inputs output values ​​of the general NeRF and performs parallel operations; a BRDF MLP into which output values ​​of the BRDF identity MLP and the Albedo MLP are input; and a renderer for performing volume rendering to generate a three-dimensional image synthesis view by rendering output values ​​of the lighting visibility MLP, the surface normal MLP, and the BRDF MLP.

[0017] It is preferable that the above machine learning view synthesis model is selected from among a plurality of machine learning view synthesis models based on the species of the tree.

[0018] According to another aspect of the present invention, a method for synthesizing a 3D view of a tree is provided, which is executed by an information processing device including a processor and a memory.

[0019] The method for synthesizing a 3D view of a tree includes the steps of collecting a plurality of multi-view 2D images of the tree taken from different directions and the species of the tree; the step of extracting tree image data by masking or deleting a background from the image to extract an area including the tree; and the step of inputting the tree image data into a machine learning view synthesis model and performing volume rendering on the output values ​​to output a 3D view.

[0020] The above machine learning view synthesis model can be selected from among a plurality of machine learning view synthesis models based on the tree species.

[0021] The above multiple machine learning view synthesis models are each trained for each tree species, and it is preferable that the machine learning view synthesis models for each tree species repeatedly train the albedo reflectance values ​​for each material of the same tree species, extract the average values ​​of the characteristics of trees that are obscured due to anisotropy or self-occlusion for each tree, and optimize them by comparing them with the anisotropy for each tree species that is input in advance.

[0022] The machine learning view synthesis model includes a general NeRF for estimating volume density at each location from a multi-viewpoint two-dimensional image; a lighting visibility multi-layer perceptron (MLP), a surface normal MLP, a BRDF ID MLP, and an Albedo MLP, each of which inputs output values ​​of the general NeRF and performs parallel computation; a BRDF MLP into which output values ​​of the BRDF ID MLP and the Albedo MLP are input; and a renderer for performing volume rendering to generate a three-dimensional image synthesis view by rendering the output values ​​of the lighting visibility MLP, the surface MLP, and the BRDF MLP.

[0023] It is preferable that the above overall NeRF and BRDF MLP are pre-learned and their determined weights are fixed, the above lighting visibility MLP (multi-layer Perceptron) and surface normal MLP are pre-learned and their weights are fine-tuned, and the above BRDF identity MLP and Albedo MLP are learned when learning the entire model.

[0024] According to another aspect of the present invention, a tree information processing system including a tree data processing device, a tree management system, and a tree transaction platform is provided.

[0025] The above tree data processing device includes a tree data processing unit that collects and processes tree data including a multi-viewpoint two-dimensional image of a tree and tree information and provides processed tree data; a communication unit that provides the processed tree data to a tree management system and a tree transaction platform through a wired or wireless communication network; and a database in which the three-dimensional view data and tree information are stored as structured data.

[0026] The tree management system is an information processing device that provides tree management information for each farm including the processed tree data, and the tree transaction platform posts the processed tree data as tree sales information and communicates with the seller device and the buyer device to relay the tree sales.

[0027] It is preferable that the tree data processing unit includes a three-dimensional view generation unit, and the three-dimensional view generation unit extracts tree image data including an area including a tree by masking or deleting a background from the multi-view two-dimensional image, inputs the tree image data into a machine learning view synthesis model, and performs volume rendering on the output values ​​to output three-dimensional view data in all directions except for the bottom of the tree.

[0028]

[0029] The present invention provides a method and system for effectively processing tree data.

[0030] The present invention also provides a method and device for constructing a machine learning view synthesis model for generating a three-dimensional image from a two-dimensional image of a tree, and providing a 3D view with excellent accuracy by removing noise.

[0031] FIG. 1 is a block diagram showing a tree data processing system according to one embodiment of the present invention.

[0032] Figure 2 is a structural diagram of a tree data processing system according to one embodiment of the present invention.

[0033] Figure 3 is a flowchart of a tree data processing method according to one embodiment of the present invention.

[0034] Figure 4 is a conceptual diagram illustrating the NeRF method for synthesizing a three-dimensional view from a two-dimensional image of a tree.

[0035] FIG. 5 is a flowchart of a method for synthesizing a three-dimensional view from a two-dimensional image of a tree according to one embodiment of the present invention.

[0036] FIG. 6 is a schematic diagram of a machine learning view synthesis model architecture according to one embodiment of the present invention.

[0037] FIG. 7 is a block diagram of a three-dimensional view generation unit of a tree data processing device according to one embodiment of the present invention.

[0038] Figure 8 is a conceptual diagram for explaining albedo and BRDF according to one embodiment of the present invention, with the left side explaining albedo and the right side explaining BRDF.

[0039]

[0040] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail. However, this does not limit the present invention to specific embodiments, but rather encompasses all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. In describing the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to unnecessarily obscure the gist of the present invention.

[0041] The terms first, second, etc. used in the description of this specification are merely identifiers to distinguish one component from another. In addition, when a component is referred to as being "connected" or "connected" to another component in this specification, it should be understood that the component may be directly connected or directly connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0042] Additionally, the singular forms used in this specification and claims should generally be construed to mean "one or more" unless otherwise stated. Furthermore, the terms "module," "part," and "interface" used herein generally refer to a computer-related object that is a unit that processes at least one function or operation, and may refer to hardware, software, or a combination thereof, for example.

[0043] In this specification, an information processing device is an electronic device including a processor (including a CPU, GPU, or a combination thereof), memory (volatile and / or non-volatile memory), input / output interface, and communication module. The information processing device may be, but is not limited to, a mobile computing device such as a smart phone, a personal computer, a laptop, a netbook, a tablet PC, a personal digital assistant (PDA), a server, a camera, etc.

[0044] Hereinafter, the present embodiment will be described in detail with reference to the attached drawings. The systems and methods described below are achieved through communication between devices such as a server and a user terminal via a communication network.

[0045] Hereinafter, the present embodiment will be described in detail with reference to the attached drawings. The systems and methods described below are implemented through wired or wireless communication between measuring devices such as servers, cameras, lidar sensors, and spectroscopic cameras and devices such as user terminals via a communication network, but are not limited thereto.

[0046] A tree information processing system according to an embodiment of the present invention is described with reference to FIGS. 1 and 2.

[0047] FIG. 1 is a schematic diagram showing the structure of a tree information processing system according to an embodiment of the present invention, and FIG. 2 is a block diagram of the tree information processing system.

[0048] The tree information processing system of FIG. 1 includes a tree data processing device (100), a learning device (300), a tree management system (500), and a tree transaction platform (600). The tree data processing system may further include a first device (120), a second device (130), an external device (140), and a tree measurement sensor (400). The first device (120) and the second device (130) may be communication-capable information processing devices used by tree owners and tree buyers, respectively, as user terminals, such as smartphones, but are not limited thereto. The external device (140) may be a device of a service provider providing services such as tree transplantation, and the tree measurement sensor (400) includes an imaging device such as a lidar sensor, a spectroscopic camera, or a camera.

[0049] The tree data processing device (100), the learning device (300), the tree management system (500), the tree transaction platform (600), the first device (120), the second device (130), and the external device (140) can communicate via a wired or wireless communication network. The tree measurement sensor (400) can also be connected to the communication network, but measurement data can also be stored in a separate memory and connected to the tree data processing device to be input.

[0050] The above tree data processing device (100) collects one or more of a two-dimensional image of a tree acquired by a first device or a camera, tree scan lidar data acquired by a lidar sensor, a tree spectral image captured by a spectral camera, and tree information input through the first device. The tree information includes the tree species and may additionally include tree specifications (height, diameter at breast height, etc.). The tree information may also include a tree image, location, tree species, specifications, sales price, training information, and tree shape. The tree information may further include the name of the farm to which the tree belongs and seller information.

[0051] The collected two-dimensional images, tree scan lidar data, tree spectral images, and tree information are processed in the tree data processing unit (101) and provided as processed data to the tree management system or tree transaction platform.

[0052] The tree data processing device (100) includes a tree data processing unit (101), a communication unit, a database, a tree inventory management unit, a learning unit, and a control unit that controls these components as a whole. The database includes a raw database that structures and stores raw tree data, a farm database, and a tree information database. The farm database stores the manager of the farm where the tree is located, the location, a list of trees within the farm, etc., and the tree information database stores tree information, specifications, tree shapes, and 3D composite view data.

[0053] Meanwhile, when a tree transaction occurs through the tree trading platform, the tree inventory management unit of the tree data processing device reflects the transaction in real time in the database. This result is then reflected in the tree management system by the tree inventory management unit.

[0054] Referring to Fig. 2, the tree data processing unit (101) and the tree data processing method are described.

[0055] The tree data processing unit (101) includes at least one of a tree object classification unit, a tree shape analysis unit, a tree standardization unit, and a 3D view generation unit. The tree object classification unit extracts the collected 2D images of trees, tree scan lidar data, and tree spectral images into individual tree area images. The tree object classification unit may be embedded in a camera, sensor, or the like rather than a tree data device. The tree object classification unit may be a tree detection model based on machine learning.

[0056] The tree shape analysis unit uses one or more of tree scan lidar data, tree shape spectral images, and two-dimensional tree image data to create a three-dimensional model of the tree and detect the tree shape from it. The tree shape analysis unit, for example, is a supervised learning classification model. When tree data is input, it outputs tree shapes, such as single-stemmed, double-stemmed, and multi-stemmed.

[0057] The tree standardization unit derives the tree standard from the tree data. The standard is the size of the tree, including the tree height (H), crown width (W), and diameter (R, B). The tree standardization unit constructs a 3D model of the tree using the individual tree data extracted by the tree object classification unit and the tree standardization model. The tree standardization model is a tree standard extraction model generated by machine learning using one or more of a 2D image of the tree, tree scan lidar data, and tree spectral image. The tree standard extraction model may include a CNN (convolutional neural network) structure, and the input value of the learning data is one or more of a 2D image of the tree, tree scan lidar data, and tree spectral image, and the output value is the standard of the corresponding tree information.

[0058] According to an embodiment of the present invention (see FIG. 3), it is also possible to analyze the tree shape using the 3D synthetic view data of the tree generated by the 3D view generation unit. In this case, the tree shape analysis can be performed using the 3D synthetic view data of the tree and the tree shape analysis model. It is also possible to calculate the tree standard using the 3D synthetic view data of the tree generated by the 3D view generation unit. In this case, the tree standard analysis can be performed using the 3D synthetic view data of the tree and the tree shape standardization model.

[0059] Referring to FIG. 3, a method for processing tree data by a tree data processing unit (101) according to one embodiment of the present invention includes a tree data collection step, an individual tree data separation step, and a 3D composite view generation step of the tree. The method may further include a step of analyzing the tree shape using the generated 3D composite view data and / or a standard extraction step.

[0060] Hereinafter, the NeRF method for synthesizing a 3D view from a 2D image of a tree is described with reference to Fig. 4. NeRF is one of the methods designed for view synthesis, and is a method for creating a function that infers the color (c) seen when looking at a given point (x, y, z) in space from a specific angle (θ, φ), the density (σ) of the point, or the distance (d) to the point through learning.

[0061] Figure 4 is a conceptual diagram illustrating the NeRF method for synthesizing a 3D view from a 2D image of a tree. Below, the specific density inference method NeRF's learning process is described.

[0062] 1. Select an imaginary line segment connecting the center of focus of each image and a point on the imaging plane.

[0063] 2. Randomly sample (multiple) locations on the above line segment. The pixel values ​​of these sampled points become the input values ​​R(x, y, z) of the function to be created in learning, and the direction of the camera focus center from these points becomes the angle (θ, φ).

[0064] 3. By inputting the sampled points R(x, y, z) and the view direction (θ, φ), the color and volume density are inferred using a neural network that infers the color and spatial density (volume density) of the corresponding point.

[0065] Ray-casting Volume Rendering is performed using the colors and densities of sampling points. This method calculates the color on the imaging plane when a given optical path is formed by accumulating the inferred colors of the corresponding sampling points.

[0066] Neural network learning is performed using a loss function based on the color difference between the calculated color and the actual surface.

[0067] By performing learning in the above manner, when looking at space from a specific location, you can infer the points with high density and the color of those points on the optical path of the points (pixels) formed on the surface. Therefore, when composing a photograph based on a neural network trained in this way, you can perform color / density (distance) inference based on the optical path for all pixels, and based on this, you can infer the image that will be formed when shooting the space from a completely new location (View Synthesis), or you can draw a Depthmap that calculates the distance to a specific object in space when shooting from that location.

[0068] For more specific details, see the paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis by Mildenhall et al. (arXiv:2003.08934v1).

[0069] However, this NeRF method infers the volume density using the first half NeRF as shown in Fig. 4, and infers the color at the corresponding coordinates according to the view direction (θ, φ) in the second half NeRF.

[0070] The above method generates an image based on an arbitrary spatial location and orientation from multiple two-dimensional images. However, this method first extracts density and then estimates color by inputting the view direction, and does not consider the material or reflectivity of the object.

[0071] Because training was performed using only RGB values ​​from the photographic data, it was difficult to identify the bark material of the tree, making it difficult to assess its growth and health. Furthermore, the anisotropic shape of the leaves resulted in high noise in the inference results and difficulty in identifying branch shapes, reducing the utility of the 3D data.

[0072] Below, we propose a new machine learning model that considers the material and reflectivity of trees to address the self-occlusion and anisotropy problems of trees.

[0073] FIG. 5 is a flowchart of a method for synthesizing a three-dimensional view from a two-dimensional image of a tree according to one embodiment of the present invention, and FIG. 6 is a schematic diagram of a machine learning view synthesis model architecture according to one embodiment of the present invention.

[0074] According to Fig. 5, tree data is first collected. The tree data includes multi-view 2D image data of the tree captured from multiple directions.

[0075] Tree data can also include tree scan lidar data and tree spectral images. Tree scan lidar data can be used to build a 3D model of a tree, and when used as training data alongside multi-view 2D image data, this allows for more accurate view composition.

[0076] According to Fig. 5, the method for synthesizing a 3D view includes a step of estimating a camera pose and eigenvalues ​​for each image, a preprocessing step, and a 3D view synthesis step using a NeRFactor model.

[0077] To synthesize a 3D view, the camera pose and eigenvalues ​​are estimated for each image of the collected tree data. For more accurate positional information on each leaf and stem, the camera pose and eigenvalues ​​for each image are estimated using Structure from Motion (SfM) technology. SFM (Structure from Motion) is the process of reconstructing 3D structure and camera pose from multi-view images of the same object taken from different viewpoints and overlapped. SfM involves the steps of feature detection and extraction, feature matching and geometric verification, and structure and motion reconstruction. Through SfM, the camera pose for each image is estimated, which can be used for the machine learning described below.

[0078] The preprocessing step extracts only individual tree regions from the image. The boundaries between the tree and the background can be extracted from the image, but if there are no leaves, the tree is extracted as a rectangular shape (depth estimation is still possible in this case).

[0079] Using semantic segmentation, we identify a rectangular box containing a tree region as the tree region in the image and mask the background. This is to reduce data processing capacity.

[0080] A machine learning view synthesis model for synthesizing a three-dimensional view from two-dimensional image data is described with reference to the architecture of FIG. 6.

[0081] According to FIG. 6, the architecture of the machine learning view synthesis model includes a general NeRF for estimating volume density at each location from multiple multi-viewpoint two-dimensional images; a lighting visibility multi-layer Perceptron (MLP), a surface normal MLP, a BRDF (Bidirectional reflectance distribution function) ID (identity) MLP, and an Albedo MLP, each of which inputs the output values ​​of the general NeRF and performs parallel computation; a BRDF MLP that inputs the output values ​​of the BRDF ID MLP and the output values ​​of the Albedo MLP; and a renderer for performing volume rendering to generate a three-dimensional image synthesis view by rendering the output values ​​of the lighting visibility MLP, the surface normal MLP, and the BRDF MLP (Bidirectional reflectance distribution function multi-layer Perceptron, bidirectional reflectance distribution function MLP).

[0082] The inputs to the entire model are assumed to be only multi-view images of objects (and their camera poses) illuminated by a single unknown light source.

[0083] To "describe" a set of input images, the shape and spatially varying reflectance of an object are represented by a series of 3D fields, each parameterized by a multi-layer perceptron (MLP) with optimized weights. After optimization, the model describes the surface normal (n) at each 3D location (x) on the object surface, the light visibility v(ω) in all directions, and the i ), albedo α and reflectance z BRDF By recovering the shape and reflectance of the object, the model enables applications such as free-viewpoint relighting (including shadows) and material editing.

[0084] The above overall NeRF and BRDF MLP are pre-learned and their weights are determined and fixed, and the light visibility MLP and surface normal MLP are pre-learned and their weights are determined, but the weights can be fine-tuned during the entire view synthesis model learning process.

[0085] Meanwhile, the 3D view data may include 3D view data from any solid angle except for the tree base. For example, the 3D view data may include view data from any direction in the upper hemisphere of the solid angle 2π (sr), which is the upper half of the total solid angle 4π (sr).

[0086] According to Fig. 6, first, the first half of NeRF (NeRF) is pre-trained using the pixel values ​​(x) of the multi-view tree image. At this time, the input values ​​of the training data are the pixel values ​​of the image data, and the output values ​​are the volume densities at all 3D positions. Therefore, the volume densities for the 3D view are generated. This process is the same as the general method of training the 1st half of NeRF. Afterwards, the output values ​​(x) on the surface of the target object, the tree, are surf ) are input into four parallel MLPs (illumination visibility MLP, surface normal MLP, albedo MLP BRDF Identity MLP).

[0087] Additionally, tree-scan lidar data can be used as an auxiliary tool to optimize tree morphology. Specifically, by optimizing the overall NeRF output during pre-training using a loss function that compares it with the data from a 3D model constructed using tree-scan lidar data, a more accurate surface shape can be obtained.

[0088] The BRDF MLP is pre-trained and its weights are determined. The input values ​​are data that NeRF has trained once, including the albedo value (α) and the potential BRDF identifier (z). BRDF) and the output value is data that contains the latent space in the shape of NeRF.

[0089] Reflectance or albedo is a unit that indicates the degree to which an object reflects light. It is calculated as the amount of reflected electromagnetic waves, and is generally expressed from 0% to 100%.

[0090] The color component of the albedo model is assumed to be processed by the albedo prediction network, which converts the RGB albedo values ​​of the MERL dataset to achromatic albedo values, discarding all color information. Potential BRDF identifier (z BRDF ) is parameterized by an unconstrained 3D vector and initialized with a zero-mean isotropic Gaussian distribution with standard deviation 0.01. The latent BRDF identifier (z BRDF ) is not penalized by sparsity or norm during training. After this pre-training, the weights of this BRDF MLP are fixed during the joint optimization process of the entire model to train the BRDF ID MLP from 0. From this, the BRDF, which predicts the spatial variation of every surface point in the reliable space of the real BRDF, optimizes the BRDF ID MLP to minimize re-rendering loss and maintain the same spatial smoothness as the hyperparameter albedo set to 0.01.

[0091] The lighting visibility MLP and the surface normal MLP are also pre-trained, and the training data is the output value of the first half of NeRF (NeRF, 1st half) on the surface of the tree (x surf ) as input, and the output is Pseudo_GT, which is the illumination visibility and surface normal of the original image. Pseudo_GT usually represents data that is artificially labeled by a computer.

[0092] Finally, the learning of the entire tree 3D view synthesis model according to the architecture of Fig. 6 is done by inputting a re-view tree image into the pre-trained overall NeRF, and selecting the output value (x) on the surface of the tree among the output values. surf ) are input to four parallel MLPs (illumination visibility MLP, surface normal MLP, albedo MLP BRDF Identity MLP), and among the output values, the output value of the albedo MLP and the output value of the BRDF ID MLP (BRDF Identity MLP) and the incident and output (ω) based on each surface normal are input. i , ω o ) angle (φ) d, θ h , θ d ) is input to the BRDF MLP. The output value of the BRDF MLP, the surface normal (n), and the light visibility v (ω) i ) is input to the renderer and rendered. The rendering result can be transformed to produce different shadows depending on the lighting.

[0093] As described above, surface normal maps and lighting visibility can be pre-trained by capturing multi-directional images of an object. BRDF and Albedo values ​​are unique to trees and leaves and vary depending on material and reflectance, requiring continuous training of reflectances for each tree species. Afterwards, tree-specific lightmaps can be applied to obtain tree data with self-occlusion removed.

[0094] That is, the above learning method continuously learns the albedo reflectance value for each material of the same tree species, and also extracts the average value of the characteristics of trees that are obscured due to anisotropy and self-occlusion for each tree, compares it with the anisotropy of each tree species that has been input in advance, and continuously performs optimization. The characteristics of the tree include bark material, leaf shape, and color value, and the average value of these characteristics is the average of the output value which is the learning result of the entire model. For example, it may be the average color value of the area corresponding to the bark. Optimization can improve accuracy by using a loss function, preferably a loss function that compares spectral data and BRDF and albedo values, but is not limited thereto.

[0095] Meanwhile, when explaining the rendering process, an image is rendered including the trained data.

[0096] a. Image data sampling

[0097] b. Extract color and density from each sampled point through NeRF model evaluation

[0098] c. Rendering extraction considering lighting visibility considering body occlusion

[0099] When the above view synthesis model is generated for each tree species, multiple 3D models are stored as shown in Fig. 7. The 3D view synthesis unit includes a 3D model determination unit that selects a view synthesis model according to the tree species, and multiple 3D (view synthesis) models.

[0100] The inferencing process is described below.

[0101] Hereinafter, a method for generating a 3D composite view using a 3D view synthesis model based on a multi-view tree image and tree species will be described. After a preprocessing step of a multi-view tree image, a 3D model determination unit determines a corresponding 3D view synthesis model among a plurality of 3D view synthesis models based on the tree species. A 3D view is generated by inputting tree image data into the determined 3D view synthesis model.

[0102] The view synthesis model of the present invention described above continuously learns the albedo reflectance value of each tree material, and also extracts the average value of the characteristics of trees that are obscured due to anisotropy and self-occlusion for each tree, and compares it with the anisotropy of each tree species input in advance to continuously perform optimization, thereby enabling view synthesis that reflects the characteristics of each tree species.

[0103] The technical contents described above may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiments, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0104] Self-occluding objects like trees can be challenging due to their three-dimensional geometry and anisotropic shape. However, applying the NeRF algorithm and incorporating additional techniques can yield better results.

[0105] The present invention has industrial applicability as it can be used in the tree-related industrial field including tree management and trading as part of tree information processing for tree trading, management, etc.

Claims

1. A tree data processing device that includes a processor and memory and processes collected tree data, A tree data processing unit that processes the above tree data and provides processed data; and including a database; The above tree data includes a two-dimensional image of the tree at multiple points, The tree data processing unit includes a 3D view generation unit, A tree data processing device characterized in that the above 3D view generation unit extracts tree image data including an area including a tree by masking or deleting a background from the re-viewpoint 2D image, inputs the tree image data into a machine learning view synthesis model, and performs volume rendering on the output values ​​to output 3D view data.

2. In paragraph 1, The above tree data processing unit further includes at least one of a tree object classification unit, a tree shape analysis unit, and a tree standardization unit, The above tree data further includes tree scan lidar data or tree spectral images, The above tree object classification unit extracts each of the collected two-dimensional images of trees, tree scan lidar data, or tree spectral images into individual tree area images. A tree data processing device characterized in that the above-mentioned tree shape analysis unit detects the tree shape using the above-mentioned three-dimensional view data.

3. In paragraph 1 or 2, The above 3D view data is tree view data generated by the machine learning view synthesis model, A tree data processing device characterized by further including a tree standardization unit that calculates the tree standard using the above three-dimensional view data.

4. In any one of paragraphs 1 to 3, The above tree data includes tree scan lidar data, The above machine learning view synthesis model includes a global NeRF that estimates volume density at each location from a two-dimensional image of a re-viewpoint, A tree data processing device characterized in that the above-mentioned overall NeRF is pre-learned and optimized using a loss function that compares the output value of the above-mentioned overall NeRF with data of a three-dimensional model constructed using tree scan lidar data.

5. In any one of paragraphs 1 to 4, The above machine learning view synthesis model is selected from among a plurality of machine learning view synthesis models based on the tree species, The above multiple machine learning view synthesis models are learned for each tree species, and the machine learning view synthesis models for each tree species repeatedly learn the albedo reflectance values ​​for each material of the same tree species, and extract the characteristics of trees that are obscured due to anisotropy or self-occlusion for each tree as an average value and compare them with the corresponding values ​​for each tree species that are input in advance to optimize them. A tree data processing device characterized in that the characteristics of the tree include at least one of bark material, leaf shape, and color value, and the average value is an average of output values ​​of a machine learning view synthesis model.

6. In any one of paragraphs 1 to 5, The above machine learning view synthesis model is, A tree data processing device characterized in that it further includes a lighting visibility MLP (multi-layer Perceptron), a surface normal MLP, a BRDF identity MLP, and an Albedo MLP, each of which inputs output values ​​of the overall NeRF and performs parallel operations; a BRDF MLP into which output values ​​of the BRDF identity MLP and the output values ​​of the Albedo MLP are input; and a renderer that performs volume rendering to generate a three-dimensional image composite view by rendering the output values ​​of the lighting visibility MLP, the surface normal MLP, and the BRDF MLP.

7. In paragraph 6, The above NeRF and BRDF MLP are pre-trained and the determined weights are fixed. The above lighting visibility MLP (multi-layer Perceptron) and surface normal MLP are pre-trained and their weights are fine-tuned during training of the entire model. The above BRDF ID MLP and Albedo MLP are learned during training of the entire model. A 3D view synthesis method characterized in that the optimization of the weights of the entire model is performed using a loss function that compares the spectral data of the tree and the BRDF and albedo values.

8. A method for synthesizing a three-dimensional view of a tree executed by an information processing device including a processor and memory, A step of collecting multiple two-dimensional images of a tree taken from different directions and the species of the tree; A step of extracting tree image data by masking or deleting the background in the image to extract an area containing trees; and A method for synthesizing a 3D view of a tree, characterized by including a step of inputting the above tree image data into a machine learning view synthesis model and performing volume rendering on the output values ​​to output 3D view data.

9. In paragraph 8, The above machine learning view synthesis model is selected from among multiple machine learning view synthesis models based on the tree species, The above multiple machine learning view synthesis models are learned for each tree species, and the machine learning view synthesis models for each tree species repeatedly learn the albedo reflectance values ​​for each material of the same tree species, and extract the characteristics of trees that are obscured due to anisotropy or self-occlusion for each tree as an average value and compare them with the corresponding values ​​for each tree species that are input in advance to optimize them. A method for synthesizing a 3D view of a tree, wherein the characteristics of the tree include at least one of bark material, leaf shape, and color value, and the average value is an average of output values ​​of a machine learning view synthesis model.

10. In paragraph 8 or 9, The 3D view data is 3D view data of a tree generated by the machine learning view synthesis model. A step of calculating the shape of a tree using the above 3D view data, A 3D view synthesis method, characterized in that it further includes a step of calculating the specifications of a tree using the above 3D view data.

11. In paragraph 8 or 9, The above machine learning view synthesis model includes a global NeRF that estimates volume density at each location from a two-dimensional image of a re-viewpoint, A 3D view synthesis method characterized in that the above-mentioned overall NeRF is pre-learned and the overall NeRF is optimized using a loss function that compares the output value of the above-mentioned overall NeRF with data of a 3D model constructed using tree scan lidar data.

12. In paragraph 11, The above-mentioned overall NeRF output values ​​are inputted respectively and parallel operations are performed on a lighting visibility MLP (multi-layer Perceptron), a surface normal MLP, a BRDF ID MLP, and an Albedo MLP; a BRDF MLP into which the output values ​​of the BRDF ID MLP and the output values ​​of the Albedo MLP are inputted; and a renderer that performs volume rendering to generate a three-dimensional image composite view by rendering the output values ​​of the lighting visibility MLP, the surface MLP, and the BRDF MLP. The above NeRF and BRDF MLP are pre-trained and the determined weights are fixed. The above lighting visibility MLP (multi-layer Perceptron) and surface normal MLP are pre-trained and their weights are fine-tuned during training of the entire model. The above BRDF ID MLP and Albedo MLP are learned during training of the entire model. A 3D view synthesis method characterized in that the optimization of the weights of the entire model is performed using a loss function that compares the spectral data of the tree and the BRDF and albedo values.

13. A tree information processing system including a tree data processing device, The tree data processing device is, A tree data processing unit that collects and processes tree data including a two-dimensional image of a tree and tree information and provides processed tree data; and A communication unit for providing the processed tree data to a tree management system and a tree trading platform through a wired or wireless communication network; The tree data processing unit includes a 3D view generation unit, The above 3D view generation unit extracts tree image data including an area containing trees by masking or deleting the background from the above multi-view 2D image, inputs the tree image data into a machine learning view synthesis model, performs volume rendering on the output values, and outputs 3D view data.

14. In paragraph 13, The above machine learning view synthesis model includes a global NeRF that estimates volume density at each location from the re-viewpoint 2D image, A tree information processing system characterized in that the above-mentioned overall NeRF is pre-learned and the overall NeRF is optimized using a loss function that compares the overall NeRF output value with data of a three-dimensional model constructed using tree scan lidar data.

15. In paragraph 13 or 14, Including a tree management system and a tree trading platform, The tree management system is an information processing device that provides tree management information for each farm including the processed tree data. The tree trading platform posts the processed tree data as tree sales information and communicates with the seller device and buyer device to relay the tree sales. The above machine learning view synthesis model is selected from among a plurality of machine learning view synthesis models based on the tree species, The above multiple machine learning view synthesis models are learned for each tree species, and the machine learning view synthesis models for each tree species repeatedly learn the albedo reflectance values ​​for each material of the same tree species, and extract the characteristics of trees that are obscured due to anisotropy or self-occlusion for each tree as an average value and compare them with the corresponding values ​​for each tree species that are input in advance to optimize them. A tree information processing system characterized in that the characteristics of the above tree include bark material, leaf shape, and color values, and the average value is an average of the output values ​​of a machine learning view synthesis model.

Citation Information

Patent Citations

  • Image processing method and electronic equipment

    CN115115688A

  • Method and apparatus for providing three-dimensional model service of street tree information

    KR1020150009324A

  • Gimbal type injection device

    KR1020230109197A

  • Tree species detection system based on LiDAR and RGB camera and Detection method of the same

    KR102308456B1

  • KR20190036435A