A multi-modal fusion three-dimensional fine modeling method for high-density orchard environment

Abstract

This disclosure provides a multimodal fusion 3D fine modeling method for high-density orchard environments, relating to the field of smart agriculture technology. The method includes: synchronously collecting multimodal data of the target orchard and aligning the multimodal data; dynamically assigning weights to the multimodal data based on real-time ambient light intensity, and performing weighted fusion based on these weights to generate basic 3D geometric data of the target orchard; performing multi-level filtering on the basic 3D geometric data and extracting the central skeleton axis; constructing a neural radiation field model, wherein the loss function of the neural radiation field model includes photometric loss and geometric constraint loss, with the geometric constraint loss calculated based on the central skeleton axis; employing a semantic topology-guided octree structure and cascaded network to accelerate rendering, calculating the photometric loss of the neural radiation field model, optimizing the neural radiation field model with the goal of minimizing the loss function, and outputting a high-precision 3D model of the orchard to support the autonomous operation of agricultural robots.

Description

Technical Field

[0001] This invention relates to the field of smart agriculture technology, and in particular to a multimodal fusion three-dimensional fine modeling method for high-density orchard environments. Background Technology

[0002] With the development of smart agriculture, digital 3D models of fruit trees are fundamental for achieving intelligent pruning, precise spraying, and yield prediction. Currently, 3D modeling of fruit trees mainly relies on the following technologies: first, LiDAR-based point cloud acquisition through laser ranging; second, vision-based methods (such as Photogrammetry / RGB-D) reconstruction using multi-view images or depth cameras; and third, Neural Radiance Fields (NeRF), which uses neural networks to fit the volume density and color of a scene, synthesizing high-quality images from new perspectives. Existing technologies typically involve rigidly registering and fusing LiDAR and RGB-D point clouds, or directly training fruit trees using a general-purpose NeRF network.

[0003] However, existing technologies have significant limitations: First, they have poor environmental adaptability. For example, LiDAR technology suffers from sparse point clouds and lacks texture information in the small branches of fruit trees, making it difficult to distinguish between lesions or bark textures. RGB-D cameras are prone to losing depth data or generating noise in orchards with strong backlighting or deep shadows, and visual algorithms easily fail on smooth, textureless tree branches. Furthermore, existing fusion methods typically use fixed weights, making it difficult to dynamically adjust according to the complex lighting conditions in orchards. Second, under sparse viewpoint input, the NeRF method easily produces cloud-like artifacts in the generated geometric models, causing branches to appear thicker or false geometry to appear, severely affecting the obstacle avoidance path planning of the robotic arm. Finally, traditional NeRF rendering is computationally intensive, while existing LOD (Levels of Detail) acceleration only reduces resolution based on distance, failing to distinguish between critical task targets (such as small pruning points) and secondary background (such as leaves). This results in blurred important areas at a distance and excessive computational consumption in non-critical areas at close range, making it impossible to guarantee the reconstruction accuracy of task targets with limited resources.

[0004] In summary, there is an urgent need to address the following issues: in unstructured, high-density orchard environments, the poor environmental adaptability of single or fixed-weight fusion sensors leads to low modeling accuracy and data loss; the NeRF algorithm is prone to producing cloud-like artifacts under sparse viewpoint input, resulting in geometric distortion that fails to meet the requirements of obstacle avoidance and precision operation for robotic arms; and traditional rendering strategies have unreasonable computing power allocation, making it difficult to achieve targeted, high-precision, real-time modeling on embedded platforms. Summary of the Invention

[0005] In view of the many shortcomings of existing technologies, this disclosure provides a multimodal fusion three-dimensional fine modeling method for high-density orchard environments. By introducing a radiative transfer physics model to simulate and analyze different atmospheric and cloud scenarios, a physical feature library that can characterize the differences between clouds and clear skies is constructed. Based on this, feature selection and optimization are performed on satellite hyperspectral observation data. Subsequently, machine learning or statistical learning models are used to model and analyze the selected features, realizing cloud probability inversion and cloud mask generation for observed pixels. This method combines the advantages of physical mechanisms and data-driven approaches, achieving high-precision, strong stability and good generalization ability for cloud detection.

[0006] The first aspect of this disclosure provides a multimodal fusion three-dimensional fine modeling method for high-density orchard environments, comprising the following steps: synchronously acquiring multimodal data of the target orchard and aligning the multimodal data; dynamically assigning weights to the multimodal data according to the real-time ambient light intensity, and performing weighted fusion of the multimodal data based on the weights to generate basic three-dimensional geometric data of the target orchard; performing multi-level filtering on the basic three-dimensional geometric data to remove dynamic outliers and retain the main structure of the fruit trees, obtaining three-dimensional geometric data representing the skeleton of the fruit trees, and extracting the central skeleton axis from the three-dimensional geometric data representing the skeleton of the fruit trees; constructing a neural radiation field model based on the basic three-dimensional geometric data, wherein the loss function of the neural radiation field model includes photometric loss and geometric constraint loss, and the geometric constraint loss is calculated based on the central skeleton axis; using a semantic topology-guided octree structure and cascaded network acceleration rendering strategy to render the neural radiation field model, and calculating the photometric loss based on the rendering results, optimizing the neural radiation field model with the goal of minimizing the loss function, and outputting a high-precision three-dimensional model of the fruit trees.

[0007] Furthermore, in some embodiments, the synchronous acquisition of multimodal data of the target orchard and the alignment processing of the multimodal data include: using a synchronization signal generator to generate a hardware synchronization trigger signal to simultaneously control the lidar, vision sensor and illumination sensor to acquire data of the target orchard, respectively obtaining lidar data, vision sensor data and ambient light intensity, wherein the vision sensor data includes real images from multiple perspectives and their corresponding depth information; converting the vision sensor data into a visual point cloud, and performing time alignment processing on the asynchronous frames in the lidar data and vision sensor data using a linear interpolation method to obtain aligned lidar point cloud and visual point cloud.

[0008] In some embodiments, the weights of multimodal data are dynamically assigned based on real-time ambient light intensity, and the multimodal data are weighted and fused based on the weights to generate basic three-dimensional geometric data of the target orchard. This includes: acquiring ambient light intensity measured by a light sensor; setting an optimal light intensity threshold and constructing a weight model using a bell-shaped function, wherein the weight model is configured such that when the ambient light intensity approaches the optimal light intensity threshold, the weight of the visual point cloud approaches 1, and the weight range of the visual point cloud is 0 to 1; assigning weights to the lidar point cloud and the visual point cloud based on the currently measured ambient light intensity and the weight model, wherein the weight of the lidar point cloud is 1 minus the weight of the visual point cloud; and weighting and fusing the lidar point cloud and the visual point cloud based on the weights to generate basic three-dimensional point cloud of the target orchard.

[0009] In some embodiments, multi-level filtering is performed on the basic three-dimensional geometric data to remove dynamic outliers and retain the main structure of the fruit tree, thereby obtaining three-dimensional geometric data characterizing the fruit tree skeleton. The central skeleton axis is then extracted from the three-dimensional geometric data characterizing the fruit tree skeleton. This includes: performing a first-level filtering on the basic three-dimensional point cloud using a density-based clustering method to remove sparse outliers that cannot be clustered; performing a second-level filtering on the basic three-dimensional point cloud using a random sampling consensus algorithm to fit the branch geometric model and remove noise points, thereby obtaining the fruit tree skeleton point cloud; and extracting the central skeleton axis of the branches from the fruit tree skeleton point cloud using Laplacian contraction or median axis extraction methods, and estimating the branch radius.

[0010] In some embodiments, the geometric constraint loss is configured such that the loss value of the sampling point is positively correlated with the volume density predicted by the model and with the surface distance to the nearest branch, and the loss value is 0 if the surface distance is negative, wherein the surface distance is the distance from the spatial sampling point to the nearest central skeleton axis minus the branch radius corresponding to the central skeleton axis.

[0011] In some embodiments, a semantic topology-guided octree structure and cascaded network accelerated rendering strategy are used to render the neural radiation field model. This includes: using a lightweight semantic segmentation network to perform semantic segmentation on the basic 3D point cloud, dividing the spatial voxels into key regions, structural regions, and background regions according to semantic importance from high to low; constructing an octree based on the semantic segmentation results using a non-uniform subdivision strategy, wherein the non-uniform subdivision strategy includes: subdividing the octree to the maximum depth in the key region, dynamically adjusting the subdivision depth of the octree according to distance in the structural region, and limiting the subdivision depth of the octree based on a preset volume density threshold in the background region; and employing a multi-level detail hierarchical rendering strategy, calling the neural radiation field model, and dynamically selecting cascaded neural networks of different complexities for rendering according to the region to which the leaf nodes of the octree belong, wherein the higher the semantic importance of the region, the higher the complexity of the selected cascaded neural network.

[0012] The second aspect of this disclosure provides a multimodal fusion 3D fine modeling system for high-density orchard environments, comprising: a data acquisition module for synchronously acquiring multimodal data of the target orchard and aligning the multimodal data; a weighted fusion module for dynamically assigning weights to the multimodal data based on real-time ambient light intensity and performing weighted fusion of the multimodal data based on the weights to generate basic 3D geometric data of the target orchard; and a preprocessing module for performing multi-level filtering on the basic 3D geometric data to remove dynamic outliers and retain the main structure of the fruit trees, thereby obtaining a 3D geometry representing the skeleton of the fruit trees. The system extracts data and extracts the central skeleton axis from the 3D geometric data representing the fruit tree skeleton; a neural radiation field modeling module is used to construct a neural radiation field model based on the basic 3D geometric data. The loss function of the neural radiation field model includes photometric loss and geometric constraint loss, with the geometric constraint loss calculated based on the central skeleton axis; an accelerated rendering module is used to render the neural radiation field model using a semantic topology-guided octree structure and cascaded network acceleration rendering strategy, and calculates the photometric loss based on the rendering results. The neural radiation field model is optimized with the goal of minimizing the loss function, and a high-precision 3D model of the fruit tree is output.

[0013] A third aspect of this disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the aforementioned first aspect.

[0014] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the aforementioned first aspect.

[0015] The fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the first aspect described above.

[0016] Due to the adoption of the above technical solution, the beneficial effects of this disclosure include: 1. This disclosure employs a multimodal data synchronization and illumination-based dynamic weighted fusion strategy. It utilizes a bell-shaped function (such as a Gaussian function) to construct a nonlinear mapping relationship between illumination intensity and visual sensor weights. This enables automatic reduction of visual weights under excessively strong or weak illumination, and increased visual weights under suitable illumination to obtain high-precision textures. This effectively solves the failure problem of visual sensors (such as RGB-D) under strong midday light (overexposure) and weak evening light (noise). Experiments show that within an illumination range of 0–100,000 Lux, the integrity of the reconstructed model consistently remains above 95%. This addresses the problems of low modeling accuracy and data loss caused by poor environmental adaptability of single or fixed-weight fusion sensors in existing technologies.

[0017] 2. This disclosure introduces branch skeleton distance constraints in NeRF training, that is, it applies penalties to spatial points that are off-center from the skeleton center and have high volume density, eliminating the cloud-like artifacts generated by traditional NeRF and avoiding geometric distortion. The branch diameter reconstruction error is controlled within 5mm, which can accurately guide the robotic arm to perform obstacle avoidance and pruning.

[0018] 3. This disclosure employs semantic topology-guided cascaded network rendering, dynamically selecting the octree subdivision depth and the number of layers in the inference neural network based on the semantic importance of the target (critical region / background region), achieving non-uniform distribution of computing power. Compared to traditional distance-based LOD methods, under the same computing resources, the reconstruction clarity of key pruning points (buds) is improved by 3 times, while the background rendering time is reduced by 60%, thus achieving near real-time high-precision modeling of the target task on an embedded platform. Attached Figure Description

[0019] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to an embodiment of the present disclosure is shown. Figure 2 This schematic diagram illustrates the overall principle block diagram of a multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to an embodiment of the present disclosure; Figure 3 An example diagram illustrating a dynamic weighted fusion according to an embodiment of the present disclosure is shown schematically; Figure 4 A schematic diagram illustrating a geometric constraint loss function according to an embodiment of the present disclosure is shown. Figure 5 An example diagram illustrating a semantically guided cascaded network LOD rendering according to an embodiment of the present disclosure is shown. Figure 6This schematically illustrates an architecture diagram of a multimodal fusion three-dimensional fine modeling system for high-density orchard environments according to an embodiment of the present disclosure; Figure 7 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] The following description, with reference to the accompanying drawings, describes a multimodal fusion three-dimensional fine modeling method, system, and electronic device for high-density orchard environments.

[0022] Figure 1 A flowchart illustrating a multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to an embodiment of the present disclosure is shown.

[0023] like Figure 1 As shown, the method includes the following steps 101-104.

[0024] Step 101: Synchronously collect multimodal data of the target orchard and perform alignment processing on the multimodal data.

[0025] In some embodiments of this disclosure, the target orchard is a high-density orchard, and multimodal data is collected from the fruit trees in the high-density orchard. This aims to solve the problems of low modeling accuracy and data loss caused by a single sensor (LiDAR or camera) in unstructured, high-density orchard environments due to foliage obstruction, strong lighting changes (such as backlighting and deep shadows), and texture loss.

[0026] In some embodiments, step 101 includes: generating a hardware synchronization trigger signal using a synchronization signal generator to simultaneously control the lidar, vision sensor, and illumination sensor to collect data from the target orchard, thereby obtaining lidar data, vision sensor data, and ambient light intensity, wherein the vision sensor data includes real images from multiple perspectives and their corresponding depth information; converting the vision sensor data into a visual point cloud; and performing time alignment processing on the asynchronous frames in the lidar data and vision sensor data using linear interpolation to obtain aligned lidar point clouds and visual point clouds.

[0027] In some embodiments, the lidar refers to LiDAR (Light Detection and Ranging), and the visual sensor is a stereo camera. The lidar can be replaced with a solid-state radar, and the RGB-D camera can be replaced with a stereo camera; this disclosure is not limiting.

[0028] Specifically, such as Figure 2 This paper demonstrates the overall hardware architecture and data processing flow of the method described in this invention. The core system hardware includes a LiDAR, an RGB-D camera, and a light sensor. These three components are synchronized at the hardware level using a precise pulse synchronization signal (such as a PPS trigger signal) generated by a synchronization controller (such as an STM32 microcontroller), ensuring the time consistency of multimodal data acquisition. The LiDAR generates a high-precision but sparse geometric point cloud, while the RGB-D camera generates an RGB image with rich texture but susceptible to illumination effects, along with pixel depth values. The data is converted into a point cloud format to obtain the RGB-D point cloud. The two data streams are processed by a time synchronization and interpolation module, using linear interpolation to register the non-fully synchronized frames.

[0029] Step 102: Based on the real-time ambient light intensity, dynamically allocate the weights of the multimodal data, and perform weighted fusion of the multimodal data based on the weights to generate the basic three-dimensional geometric data of the target orchard.

[0030] In the embodiments of this disclosure, in order to solve the reliability problem of different sensors under different lighting conditions in related technologies, the method of the present invention adopts a dynamic weight allocation strategy, introduces a light sensor to measure the ambient light intensity, and constructs a nonlinear mapping relationship between light intensity and visual sensor weight, so that the system automatically reduces the visual weight when the light is too strong or too weak, and increases the visual weight when the light is suitable.

[0031] In some embodiments, step 102 includes: acquiring the ambient light intensity measured by a light sensor; setting an optimal light intensity threshold and constructing a weight model using a bell-shaped function, wherein the weight model is configured such that when the ambient light intensity approaches the optimal light intensity threshold, the weight of the visual point cloud approaches 1, and the weight range of the visual point cloud is 0 to 1; assigning weights to the lidar point cloud and the visual point cloud according to the currently measured ambient light intensity and the weight model, wherein the weight of the lidar point cloud is 1 minus the weight of the visual point cloud; and performing weighted fusion of the lidar point cloud and the visual point cloud based on the weights to generate a basic three-dimensional point cloud of the target orchard.

[0032] In some embodiments, such as Figure 2 and Figure 3As shown, the ambient light intensity signal collected in real time by the light sensor is input to the multimodal weighted fusion module. Based on the preset Gaussian weight model, the module calculates the confidence weight of the RGB-D data and the complementary weight of the LiDAR data under the current lighting conditions, and performs weighted fusion on the two aligned point clouds to output a basic fused point cloud with confidence information.

[0033] Specifically, RGB-D cameras perform best under suitable lighting conditions. Data reliability decreases in excessively bright (overexposure) or dim (noise) lighting, in which case LiDAR should be relied upon primarily. Introducing a light sensor to measure ambient light intensity is also beneficial. Set the optimal light intensity threshold (e.g., 1000 Lux) and bandwidth parameters The weight model is constructed using a Gaussian function, as shown in Formula 1 below: (Formula 1) in, The confidence weights for visual data (range 0~1).

[0034] Furthermore, the weights of the LiDAR data are... 1- .when near hour, Approaching 1; deviating when the light is extremely strong or weak. hour, It rapidly attenuates and automatically switches to LiDAR geometric data as the primary source. That is, when the lighting is too strong or too dark, it automatically reduces the weight of visual data and relies mainly on LiDAR geometric data; when the lighting is suitable, it increases the weight of visual data to obtain high-precision textures.

[0035] It should be noted that in specific implementations, the Gaussian function can be replaced with a piecewise linear function or a cosine function, as long as it satisfies the bell-shaped distribution characteristics of high in the middle (suitable lighting) and low at both ends (extreme lighting).

[0036] Step 103: Perform multi-level filtering on the basic three-dimensional geometric data to remove dynamic outliers and retain the main structure of the fruit tree, thereby obtaining three-dimensional geometric data that characterizes the fruit tree skeleton, and extract the central skeleton axis from the three-dimensional geometric data that characterizes the fruit tree skeleton.

[0037] In some embodiments, a two-stage filtering method is used to address dynamic noise such as flying insects and fallen leaves in the orchard, such as... Figure 2As shown, the fused point cloud is filtered in two stages using the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) density clustering algorithm and the RANSAC (Random Sample Consensus) algorithm to extract the central skeleton line. Specifically, the first stage of filtering uses the DBSCAN density clustering algorithm, setting a neighborhood radius. The minimum number of points, MinPts, is used to remove sparse outliers that cannot be clustered. DBSCAN can be replaced by Euclidean clustering algorithm, which is not restricted in this disclosure. The second-stage filtering uses the RANSAC algorithm to perform local cylindrical fitting on the trunk and thick branches, retaining the point cloud that conforms to the continuity of tree growth, and removing noise points that are too far away from the fitted model to obtain a pure fruit tree skeleton point cloud.

[0038] Furthermore, from the filtered point cloud, the central skeleton line of the fruit tree is extracted using Laplacian contraction or median axis extraction algorithms, and the approximate radius r is estimated.

[0039] Step 104: Construct a neural radiation field model based on the basic three-dimensional geometric data. The loss function of the neural radiation field model includes photometric loss and geometric constraint loss. The geometric constraint loss is calculated based on the central skeleton axis.

[0040] In some embodiments, in order to address the problem that existing Neural Radiation Field (NeRF) algorithms are prone to producing cloud-like artifacts and geometric distortions under sparse viewpoint inputs, thus failing to meet the requirements of obstacle avoidance and precision operations for robotic arms, a geometric topological prior of fruit tree growth is introduced into the network training.

[0041] In some embodiments, the NeRF model is trained using the central skeleton line as the geometric prior input and real images and depth data from multiple perspectives in the visual data as supervised input. Photometric loss is calculated by comparing the predicted image rendered by the model with the real image, and geometric constraint loss is calculated based on the central skeleton line. This geometric constraint loss is used to penalize spatial points that deviate from the skeleton center and have high volume density to eliminate geometric artifacts, resulting in a high-precision 3D model of the fruit tree.

[0042] In some embodiments, the geometric constraint loss is configured such that the loss value of the sampling point is positively correlated with the volume density predicted by the model and with the surface distance to the nearest branch, and the loss value is 0 if the surface distance is negative, wherein the surface distance is the distance from the spatial sampling point to the nearest central skeleton axis minus the branch radius corresponding to the central skeleton axis.

[0043] Specifically, such as Figure 2As shown, spatiotemporally aligned multi-view RGB images and their depths are used as the primary supervisory data. Specifically, for each camera ray to be rendered, a series of 3D spatial points are sampled within the depth range along the ray's path. For each sampled point, the input format is its spatial coordinates (x, y, z) and the viewpoint represented by the direction of that camera ray (x, y, z). The input undergoes positional encoding, and the encoded data is then fed into a multilayer perceptron (MLP, or NeRF neural network). The core function of this network is to act as a continuous scene function, predicting two values—volume density—for any point and direction in space. The predicted image is synthesized using RGB color c and a volumetric rendering process. Model training is performed by minimizing a joint loss function. Driven by backpropagation or gradient descent, network parameters are updated.

[0044] Furthermore, the joint loss function Includes photometric loss and geometric constraint loss , The modeling is shown in Equation 2: (Formula 2) in, From sampling point p to the nearest skeleton axis The distance is given by r, where r is the estimated radius of the branch at that location. The joint loss function is... , The hyperparameters used to balance the weights of the two losses can flexibly control the strength of prior geometric constraints to adapt to different scenarios or data conditions.

[0045] Figure 4 The geometric constraint loss introduced in NeRF training by this invention is explained in the form of a two-dimensional cross-sectional schematic diagram. The working principle, such as Figure 4 As shown, when the sampling point p is far from the skeleton axis distance The sampled point is located far beyond the true radius r, meaning it is in the air and does not belong to the true ground truth, and the network predicts the volume density of this sampled point. A high penalty is imposed if the point is considered to contain an object.

[0046] Step 105: Using a semantic topology-guided octree structure and cascaded network to accelerate rendering, the neural radiation field model is rendered, and the photometric loss is calculated based on the rendering results. The neural radiation field model is optimized with the goal of minimizing the loss function, and a high-precision three-dimensional model of the fruit tree is output.

[0047] It should be noted that, in order to resolve the contradiction between small target (bud point / cut) and cluttered background (leaves) in agricultural robot operations, this invention abandons the traditional distance-based LOD strategy and proposes a cascaded network-based accelerated rendering method guided by semantic topology. This method combines semantic segmentation with non-uniform rendering of octrees, and can dynamically select the octree subdivision depth and the number of layers of the inference neural network according to the semantic importance of the target.

[0048] In some embodiments, the accelerated rendering method includes: performing semantic segmentation on the basic 3D point cloud using a lightweight semantic segmentation network (such as PointNet), and dividing the spatial voxels into three categories according to semantic importance from high to low: key regions, structural regions, and background regions. Specifically, in agricultural robot pruning operations, key regions include buds and branching points, structural regions include trunks and thick branches, and background regions include leaves and the ground.

[0049] Furthermore, based on the semantic segmentation results, an octree is constructed using a non-uniform subdivision strategy. This strategy includes: in critical regions, regardless of distance from the camera, forcing the octree to subdivide to the maximum depth (e.g., leaf node size 2mm) to preserve minute features; dynamically adjusting the subdivision depth of the octree based on distance in structural regions; and limiting the subdivision depth of the octree in background regions, terminating splitting early for areas with volume density below a preset threshold or marked as leaves to prevent invalid details from occupying storage.

[0050] Furthermore, a Level of Details (LOD) cascaded inference based on network capacity is employed, dynamically switching the complexity of the inference network according to the semantic level of the octagonal leaf nodes. Specifically, such as... Figure 5 As shown, the key area, structure area, and background area correspond to three levels of detail: high-precision Level 0, medium-precision Level 1, and coarse-precision Level 2, respectively. Level 0 targets buds and pruning cuts, inputting high-frequency position codes and using a deep MLP network (e.g., 8 fully connected layers) for inference to accurately recover surface texture and cut direction. Level 1 targets the trunk, inputting low-frequency position codes and using a lightweight MLP network (e.g., 2-4 layers) for inference, recovering only the geometric shape and ignoring subtle textures. Level 3 targets leaves, without performing neural network inference, directly reading the average value of the pre-stored spherical harmonics (SH) coefficients in the octagonal leaf nodes as the color output. This cascaded strategy concentrates computational power on key pruning points.

[0051] It should be noted that this invention achieves non-uniform distribution of computing power through semantic topology-guided cascaded network rendering. Compared with traditional distance-of-destruction (LOD) methods, under the same computing resources, the reconstruction clarity of key pruning points is improved by 3 times, while the background rendering time is reduced by 60%, thus realizing near real-time high-precision modeling of the target task on an embedded platform.

[0052] Figure 6 A multimodal fusion 3D fine modeling system 200 for high-density orchard environments, according to an embodiment of the present disclosure, is illustrated schematically. Figure 6 As shown, the system includes: The data acquisition module 210 is used to synchronously acquire multimodal data of the target orchard and perform alignment processing on the multimodal data; The weighted fusion module 220 is used to dynamically allocate weights to multimodal data according to the real-time ambient light intensity, and perform weighted fusion of multimodal data based on the weights to generate basic three-dimensional geometric data of the target orchard; The preprocessing module 230 is used to perform multi-level filtering on the basic three-dimensional geometric data to remove dynamic outliers and retain the main structure of the fruit tree, thereby obtaining three-dimensional geometric data that characterizes the fruit tree skeleton, and extracting the central skeleton axis from the three-dimensional geometric data that characterizes the fruit tree skeleton. The neural radiation field modeling module 240 is used to construct a neural radiation field model based on basic three-dimensional geometric data. The loss function of the neural radiation field model includes photometric loss and geometric constraint loss. The geometric constraint loss is calculated based on the central skeleton axis. The accelerated rendering module 250 is used to render the neural radiation field model using a semantic topology-guided octree structure and cascaded network accelerated rendering strategy, and calculates the photometric loss based on the rendering results. The neural radiation field model is optimized with the goal of minimizing the loss function, and a high-precision three-dimensional model of the fruit tree is output.

[0053] It should be noted that the specific implementation methods of the embodiments disclosed herein are different from those of the embodiments in this paper. Figure 1 The principle of the embodiments shown is the same, and will not be repeated here.

[0054] Figure 7 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0055] like Figure 7 As shown, device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 302 or a computer program loaded from storage unit 308 into RAM (Random Access Memory) 303. RAM 303 can also store various programs and data required for the operation of device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O (Input / Output) interface 305 is also connected to bus 304.

[0056] Multiple components in device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of monitors, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0057] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as a multimodal fusion 3D fine modeling method for high-density orchard environments. For example, in some embodiments, the multimodal fusion 3D fine modeling method for high-density orchard environments can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by computing unit 301, one or more steps of the method described above can be performed. Alternatively, in other embodiments, computing unit 301 can be configured by any other suitable means (e.g., by means of firmware) to perform the aforementioned multimodal fusion three-dimensional fine modeling method for high-density orchard environments.

[0058] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0059] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0060] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0061] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0062] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0063] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0064] It's important to note that artificial intelligence (AI) is the study of using computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0065] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0066] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multimodal fusion three-dimensional fine modeling method for high-density orchard environments, characterized in that, The method includes: Simultaneously collect multimodal data from the target orchard and perform alignment processing on the multimodal data; Based on the real-time ambient light intensity, the weights of the multimodal data are dynamically allocated, and the multimodal data are weighted and fused based on the weights to generate the basic three-dimensional geometric data of the target orchard; The basic three-dimensional geometric data is filtered in multiple stages to remove dynamic outliers and retain the main structure of the fruit tree, thereby obtaining three-dimensional geometric data that characterizes the fruit tree skeleton, and the central skeleton axis is extracted from the three-dimensional geometric data that characterizes the fruit tree skeleton. A neural radiation field model is constructed based on the aforementioned basic three-dimensional geometric data. The loss function of the neural radiation field model includes photometric loss and geometric constraint loss, wherein the geometric constraint loss is calculated based on the central skeleton axis. A semantic topology-guided octree structure and cascaded network acceleration rendering strategy are adopted to render the neural radiation field model. The photometric loss is calculated based on the rendering results, and the neural radiation field model is optimized with the goal of minimizing the loss function, outputting a high-precision three-dimensional model of fruit trees.

2. The multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to claim 1, characterized in that, The process of synchronously acquiring multimodal data from the target orchard and aligning the multimodal data includes: A hardware synchronization trigger signal is generated using a synchronization signal generator to simultaneously control the lidar, vision sensor, and light sensor to collect data from the target orchard, obtaining lidar data, vision sensor data, and ambient light intensity, respectively. The vision sensor data includes real images from multiple perspectives and their corresponding depth information. The visual sensor data is converted into point cloud form, and the asynchronous frames in the lidar data and the visual sensor data are time-aligned using linear interpolation to obtain the aligned lidar point cloud and the visual point cloud.

3. The multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to claim 2, characterized in that, The step of dynamically assigning weights to the multimodal data based on real-time ambient light intensity, and then weighting and fusing the multimodal data based on these weights to generate the basic three-dimensional geometric data of the target orchard includes: The ambient light intensity measured by the light sensor is obtained; An optimal light intensity threshold is set, and a weight model is constructed using a bell-shaped function. The weight model is configured such that when the ambient light intensity approaches the optimal light intensity threshold, the weight of the visual point cloud approaches 1, and the weight range of the visual point cloud is 0 to 1. Based on the currently measured ambient light intensity and the weighting model, weights are assigned to the lidar point cloud and the visual point cloud, wherein the weight of the lidar point cloud is 1 minus the weight of the visual point cloud. The lidar point cloud and the visual point cloud are weighted and fused based on the weights to generate the basic three-dimensional point cloud of the target orchard.

4. The multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to claim 3, characterized in that, The process of performing multi-level filtering on the basic three-dimensional geometric data to remove dynamic outliers and retain the main structure of the fruit tree, thereby obtaining three-dimensional geometric data representing the fruit tree skeleton, and extracting the central skeleton axis from the three-dimensional geometric data representing the fruit tree skeleton, includes: A density-based clustering method is used to perform the first-level filtering on the basic 3D point cloud to remove sparse outliers that cannot be clustered. The basic 3D point cloud is filtered in a second stage using a random sampling consensus algorithm to fit the branch geometric model and remove noise points, thus obtaining the fruit tree skeleton point cloud. From the point cloud of the fruit tree skeleton, the central skeleton axis of the branches is extracted using Laplace shrinkage or median axis extraction, and the branch radius is estimated.

5. The multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to claim 4, characterized in that, The geometric constraint loss is configured as follows: The loss value of the sampling point is positively correlated with the volume density predicted by the model and with the surface distance to the nearest branch. If the surface distance is negative, the loss value is 0. The surface distance is the distance from the spatial sampling point to the nearest central skeleton axis minus the branch radius corresponding to the central skeleton axis.

6. The multimodal fusion three-dimensional fine modeling method for high-density orchard environments according to claim 5, characterized in that, The rendering of the neural radiation field model using a semantic topology-guided octree structure and cascaded network accelerated rendering strategy includes: A lightweight semantic segmentation network is used to perform semantic segmentation on the basic 3D point cloud, and the spatial voxels are divided into key regions, structural regions and background regions according to semantic importance from high to low. Based on the semantic segmentation results, an octree is constructed using a non-uniform subdivision strategy. The non-uniform subdivision strategy includes: subdividing the octree to the maximum depth in the critical region, dynamically adjusting the subdivision depth of the octree according to the distance in the structural region, and limiting the subdivision depth of the octree in the background region based on a preset volume density threshold. A multi-detail level hierarchical rendering strategy is adopted, the neural radiation field model is invoked, and cascaded neural networks of different complexities are dynamically selected for rendering based on the region to which the octagonal leaf node belongs. The higher the semantic importance of the region, the higher the complexity of the selected cascaded neural network.

7. A multimodal fusion three-dimensional fine modeling system for high-density orchard environments, characterized in that, The system includes: A data acquisition module is used to synchronously acquire multimodal data of the target orchard and perform alignment processing on the multimodal data; The weighted fusion module is used to dynamically allocate weights to the multimodal data according to the real-time ambient light intensity, and perform weighted fusion on the multimodal data based on the weights to generate the basic three-dimensional geometric data of the target orchard. The preprocessing module is used to perform multi-level filtering on the basic three-dimensional geometric data to remove dynamic outliers and retain the main structure of the fruit tree, thereby obtaining three-dimensional geometric data that characterizes the fruit tree skeleton, and extracting the central skeleton axis from the three-dimensional geometric data that characterizes the fruit tree skeleton. A neural radiation field modeling module is used to construct a neural radiation field model based on the basic three-dimensional geometric data. The loss function of the neural radiation field model includes photometric loss and geometric constraint loss, and the geometric constraint loss is calculated based on the central skeleton axis. An accelerated rendering module is used to render the neural radiation field model using a semantic topology-guided octree structure and cascaded network accelerated rendering strategy, and calculates the photometric loss based on the rendering results. The neural radiation field model is optimized with the goal of minimizing the loss function, and a high-precision three-dimensional model of the fruit tree is output.

8. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images