Conical inflorescence type fruit tree flower three-dimensional reconstruction male and female flower counting method and system

Abstract

The invention discloses a cone inflorescence type fruit tree flower three-dimensional reconstruction male and female flower counting method and system, and the method comprises the steps: S1, collecting an image sequence, shooting an inflorescence video through a camera, and extracting the video into an image frame by frame; s2, image preprocessing and camera parameter estimation: preprocessing the inflorescence image, and acquiring a camera parameter corresponding to each image by using a COLMAP tool; s3, a D-NeRF model is trained, and multi-modal three-dimensional reconstruction of the inflorescence point cloud is carried out by using the D-NeRF model; s4, the three-dimensional point cloud output by the D-NeRF model is optimized and marked; s5, training a PointNet + + model, and performing male and female flower identification and counting by using the PointNet + + model; and S6, dividing a data set, and adopting a Bayesian optimization algorithm to adjust and optimize key hyper-parameters in the training process of the D-NeRF model and the PointNet + + model. According to the invention, through integration of NeRF reconstruction, point cloud processing, deep learning target detection and mobile terminal deployment, three-dimensional reconstruction of fruit tree inflorescence and male and female flower identification and counting are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and intelligent agriculture, and particularly relates to a method and system for counting male and female flowers of panicle fruit trees through three-dimensional reconstruction. Background Art

[0002] In fruit tree cultivation and precision agricultural management, the number of flowers during the flowering period is one of the important factors affecting fruit yield. Especially for panicle-type fruit trees (such as apples and pears), accurately and efficiently counting the number of flowers is of great significance for yield prediction, fertilization management, and pest and disease control. However, traditional flower counting methods mainly rely on manual observation and target detection. Manual counting is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in insufficient accuracy of the counting results. For dense and small panicle-type fruit tree flowers, target detection not only takes a long time to detect, but is also prone to missed detections and false detections, resulting in a significant decrease in detection efficiency. Therefore, developing a flower counting method with high detection accuracy has important application value.

[0003] In recent years, with the advancement of computer vision and deep learning technologies, three-dimensional reconstruction methods based on images and point cloud data have become an important tool for plant phenotyping. For monitoring flower abundance during the flowering period of fruit trees, traditional two-dimensional image-based detection methods, limited by the imaging perspective, often suffer from issues such as sensitivity to spatial occlusion, difficulty separating overlapping flowers within the same view, and low counting accuracy due to a lack of depth information. These issues are particularly pronounced when dealing with densely packed panicles of fruit trees with complex growth patterns, severely impacting the accuracy and stability of identification and counting.

[0004] To overcome these limitations, the introduction of 3D reconstruction technology has become a key approach to improve recognition robustness and counting accuracy. In particular, the introduction of Neural Radiance Field (NeRF) technology in recent years has made it possible to reconstruct 3D structures from video data with high precision. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a method and system for three-dimensional reconstruction of panicle fruit tree flowers and counting of male and female flowers. By integrating NeRF reconstruction, point cloud processing, deep learning target detection and mobile terminal deployment, three-dimensional reconstruction of fruit tree inflorescences and identification and counting of male and female flowers are achieved.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction comprises the following steps:

[0008] S1, collect image sequences, use a camera to shoot inflorescence video, and extract the video into images frame by frame;

[0009] S2, image preprocessing and camera parameter estimation, preprocess the inflorescence images and use the COLMAP tool to obtain the camera parameters corresponding to each image;

[0010] S3, training the D-NeRF model and using the D-NeRF model to perform multimodal 3D reconstruction of inflorescence point clouds;

[0011] S4, optimize and annotate the 3D point cloud output by the D-NeRF model;

[0012] S5. Train the PointNet++ model and use it to identify and count male and female flowers.

[0013] S6. Divide the dataset and use the Bayesian optimization algorithm to tune the key hyperparameters in the training process of the D-NeRF model and PointNet++ model.

[0014] The present invention also includes a panicle fruit tree flower three-dimensional reconstruction male and female flower counting system, which adopts the counting method provided by the present invention and includes a front-end applet module and a cloud-based reasoning service module;

[0015] The front-end applet module is used for inflorescence video collection, compression and upload, recognition result display and visual interaction, processing progress query and status feedback;

[0016] The cloud-based inference service module is deployed on the cloud server and includes a video reception and preprocessing module, a D-NeRF 3D reconstruction module, a point cloud recognition and male and female flower classification module, and a result encoding and transmission module.

[0017] Furthermore, the video reception and preprocessing module uses a SpringBoot-based Web framework to build a data reception service, which receives and parses the videos uploaded by the front-end applet module; samples and extracts the videos to generate image sequences for 3D reconstruction;

[0018] The D-NeRF 3D reconstruction module calls the D-NeRF model deployed in the cloud to reconstruct the 3D point cloud structure of the inflorescence based on the collected image sequence;

[0019] The point cloud recognition and male and female flower classification module segments and preprocesses the reconstructed 3D point cloud, then inputs the trained PointNet++ model to perform male and female flower recognition at each point and output the classification results.

[0020] The result encoding and transmission module compresses and encodes the key information in the classification results, retaining only the necessary fields, and returns them to the front-end mini-program module in a lightweight JSON format;

[0021] In the cloud-based inference service module, the trained D-NeRF and PointNet++ models are exported in ONNX format and combined with the TensorRT tool for graph structure optimization, quantization, and pruning. This further compresses the model size, improves inference efficiency, reduces cloud resource consumption, and enhances the overall system's response speed and processing capabilities.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] 1. The present invention introduces the D-NeRF model for the reconstruction of three-dimensional point clouds of panicle fruit tree flowers, which greatly improves the reconstruction accuracy and detail restoration. Compared with traditional three-dimensional reconstruction methods based on structured light modeling or multi-view stereo (MVS) methods, the present invention is based on D-NeRF and can achieve high-precision three-dimensional point cloud restoration of the complex structure of the inflorescence while relying only on ordinary video sequences. In addition, by modeling the temporal deformation field, the present invention can achieve dynamic three-dimensional reconstruction, significantly improving the reconstruction robustness of the fruit tree inflorescence under non-rigid interference such as wind and insect activity, effectively solving the problems of topological breaks and geometric mismatches, and generating more realistic and complete three-dimensional point cloud data. Compared with two-dimensional image flower quantity detection, D-NeRF can model illumination changes and occlusion information in space, effectively improving the integrity and resolution of the inflorescence point cloud, and is particularly suitable for scenes such as leaf occlusion and dense flowers during the flowering period, providing a more reliable data foundation for subsequent point cloud recognition tasks.

[0024] 2. The present invention uses the PointNet++ network structure to classify male and female flowers on the high-quality point cloud reconstructed by D-NeRF. By introducing a manually annotated point cloud dataset, a block processing strategy, and a spatial clustering algorithm, it achieves point-level recognition of male and female flowers and counting of flower clusters, solving the problem that traditional image methods have difficulty in accurately identifying overlapping and dense flower clusters, and greatly improving recognition precision and counting accuracy.

[0025] 3. The present invention proposes a deployment architecture of mobile terminal applet plus cloud-based reasoning service; considering that models such as D-NeRF and PointNet++ are large in size and have high computational complexity, they are not suitable for direct deployment on mobile devices. The system of the present invention designs a system architecture of front-end lightweight acquisition plus cloud-based centralized computing plus front-end visual display; the front-end applet realizes video acquisition, compression upload and result visualization, and the cloud completes three-dimensional reconstruction and classification reasoning, which effectively reduces the terminal computing power burden and ensures that the system can run stably on mobile devices such as conventional smartphones; the present invention also supports dual modes of offline acquisition and online processing, and is compatible with both online real-time processing and offline delayed upload processing modes, and is suitable for complex outdoor environments such as farmland and orchards; in the absence of network or weak network conditions, users can complete video acquisition offline, and automatically upload it to the cloud to perform processing tasks after the device is connected to the network, which improves the applicability and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram of the method of the present invention;

[0027] Figure 2 This is a schematic diagram of the NeRF network structure;

[0028] Figure 3 This is the NeRF-COLMAP joint reconstruction flowchart;

[0029] Figure 4 Schematic diagram of the deployment of the counting system in the embodiment. DETAILED DESCRIPTION

[0030] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0031] Explanation of related terms:

[0032] NeRF: Neural Radiance Fields, neural radiation field;

[0033] D-NeRF: Dynamic Neural Radiance Fields, dynamic neural radiation field.

[0034] like Figure 1 As shown, the present invention provides a method for counting male and female flowers of panicle fruit trees through three-dimensional reconstruction, comprising the following steps:

[0035] S1. Collect image sequences, use a camera to shoot inflorescence videos, and extract the videos into images frame by frame; specifically,

[0036] Use a camera to photograph the inflorescence. Move the camera around the inflorescence and shoot from different angles, keeping the distance between the camera and the inflorescence constant. Try to cover all directions of the inflorescence from various angles, including up and down, left and right, and front and back. Keep the inflorescence in a fixed position during the shooting process. Try to shoot when the wind speed is low (less than 1.5m / s) and the light is even during the day to avoid large-scale shaking of the inflorescence, direct sunlight, or heavy shadows.

[0037] After shooting, the video is saved in MP4 format. FFmpeg is used to extract the video frame by frame into images, extracting one image every two frames to obtain a set of time-ordered image sequences. The images are saved in PNG format and finally numbered in sequence.

[0038] S2, image preprocessing and camera parameter estimation, preprocess the inflorescence images and use the COLMAP tool to obtain the camera parameters corresponding to each image; specifically, it includes:

[0039] Crop and scale the inflorescence image and unify the image size; perform color normalization on the image and scale the pixel values ​​to between 0 and 1;

[0040] Check the image and remove blurry or unsharp images caused by jitter. Use OpenCV filtering methods to clarify blurry or unevenly exposed images, such as Gaussian filtering or sharpening.

[0041] Use the COLMAP tool to obtain the camera parameters corresponding to each image, namely the camera's position (displacement) and orientation (rotation angle); input the image into COLMAP, use the Structure-from-Motion algorithm to calculate the camera's extrinsic parameters (external parameters) and intrinsic parameters (internal parameters), output the camera pose and corresponding sparse point cloud for each image, and finally save the camera pose parameters as a transform.json format file.

[0042] S3, training the D-NeRF model and using the D-NeRF model to perform multimodal 3D reconstruction of inflorescence point clouds;

[0043] This method uses NeRF model for training and generates high-precision 3D point cloud. NeRF mainly uses Multi-Layer Perceptron (MLP) for modeling. The typical network structure is as follows Figure 2 As shown in the figure; it uses implicit neural representation and volume rendering methods, which can better fit the color and volume density distribution in complex lighting and occlusion environments. This method uses dynamic neural radiation field D-NeRF to realize multimodal reconstruction of inflorescence point cloud, aiming to solve the problem of three-dimensional point cloud reconstruction error caused by dynamic deformation of plant organs in natural scenes. In view of the non-rigid motion of inflorescences in open environments caused by environmental disturbances (such as wind action, contact with pollinating insects, etc.), the traditional static NeRF model faces challenges in cross-view geometric consistency constraints due to its lack of temporal modeling capabilities, such as topological breaks caused by motion blur, the inability of rigid assumptions to describe complex deformations such as petal wrinkles and filament swings, and multi-view visibility conflicts caused by moving objects.

[0044] Step S3 specifically includes:

[0045] S31, input data, including image sequence (ordered by time), camera parameters of each image and its corresponding timestamp (e.g., t=0 for the first image and t=0.05 seconds for the second image);

[0046] S32, training D-NeRF model; D-NeRF model consists of two fully connected neural networks MLP, canonical network (Canonical Network) Ψ xand Deformation Network Ψ t ; Standardize network x , learning the static implicit representation of the scene in the canonical space (i.e., time t = 0), outputting the volume density σ and view-dependent color c of the 3D point x; deformation network Ψ t , modeling the time domain deformation field, mapping the observation space coordinates (x, t) to the standard space coordinates x′=x+Δx, where Δx=Ψ t (x,t);

[0047] The dynamic volume rendering equation calculates the color on the camera ray r(h) = o + hd as follows:

[0048]

[0049] Among them, x ‘ =r(h)+Ψ t (r(h,t)) is the canonical space coordinate after deformation, is the cumulative transmittance, σ and c are the volume density and color respectively;

[0050] During the training process, some pixels are randomly selected from the image in each round, and the difference between the color rendered by the model and the color of the original image is calculated (L2 loss), and the network weights are continuously optimized. After the training is completed, the mean square error (MSE) between the rendered pixels and the real pixels is minimized to jointly optimize the Ψ x With Ψ t The model outputs a time-varying 3D implicit field, which can generate high-resolution images at any time and angle of view and corresponding time-space consistent point cloud data through dynamic volume rendering.

[0051] S33. Render and export the 3D point cloud. The trained D-NeRF model can generate a new image from any perspective. Using the D-NeRF model, rays are sampled from multiple angles, and the density values ​​output by the model are converted into the positions of volume points. A density threshold is set, and points exceeding the density threshold are considered to be points where real objects exist. The entire 3D space is traversed, and high-density points are collected to obtain a dense set of points, namely the 3D point cloud. Finally, these points are exported as a PLY format file.

[0052] like Figure 3 Shown is the NeRF-COLMAP joint reconstruction flowchart.

[0053] S4. Optimize and annotate the 3D point cloud output by the D-NeRF model; specifically,

[0054] S41, point cloud denoising and enhancement, load the 3D point cloud (including RGB information) generated by the D-NeRF model into the CloudCompare software, and use the Noise filter to remove outliers;

[0055] S42, annotation content, manually annotating the inflorescence area (distinguishing between male and female flowers and background) in the 3D point cloud generated by the D-NeRF model;

[0056] Use the hierarchical annotation tool (Edit→ScalarFields→SetasClassification) to annotate the point cloud, including:

[0057] Semantic annotation: mark the female flowers (red), male flowers (blue), and background (gray) in the point cloud. Each point contains (x, y, z, label), label∈{female flower, male flower, background};

[0058] Instance labeling, assigns a unique ID to each flower instance to support counting tasks.

[0059] S43. Export the annotation data. Select the point cloud object in the CloudCompare software, ensure that the annotation information is saved in the file, and export it to a txt file. The file contains the coordinates (X, Y, Z), color, and label value of each point.

[0060] S5. Train the PointNet++ model and use it to identify and count male and female flowers. This includes:

[0061] S51. Preprocessing the original point cloud data; dividing the three-dimensional point cloud of the entire inflorescence into multiple local sub-point cloud blocks using a spatial partitioning method, wherein the sub-point cloud blocks contain a fixed number (preferably 1024) of points; if the number of points in a sub-block is insufficient, repeated sampling is used to fill it; if the number of points exceeds, random downsampling is used to intercept; normalizing the coordinates of all points; and retaining the original category label information for each sub-point cloud block;

[0062] S52. Train the PointNet++ model. The PointNet++ model consists of multiple layers of feature extraction modules. Each layer includes the following operations: point sampling (using the farthest point sampling algorithm), neighborhood query (building local regions based on the sphere radius), and local feature extraction (multi-layer perceptron (MLP) combined with maximum pooling operations). The model input is the point coordinate information of each sub-point cloud block (optionally with color information), and the output is the category probability of each point. During training, the cross-entropy loss function is used for error backpropagation, the Adam optimizer is used, the learning rate is set to 0.001, and the number of iterations is determined by the accuracy of the validation set, which is usually more than 100 rounds.

[0063] S53. After training, the PointNet++ model is used to predict the reconstructed inflorescence 3D point cloud data. Specifically:

[0064] First, the complete point cloud is divided into several sub-point cloud blocks according to the same block division method as in training. These blocks are then input into the trained PointNet++ model to obtain the classification probability of each point. Finally, a predicted label is assigned to each point to achieve the classification of male and female flower points.

[0065] S54, counting the male and female flowers based on the male and female flower identification results, specifically:

[0066] The point clouds classified as female flowers and male flowers were extracted into two independent subsets respectively. The density-based spatial clustering algorithm DBSCAN was applied to the two sub-point cloud sets. The algorithm does not require a preset number of clusters and can identify the distribution of point clusters in natural growth structures. By counting the number of clusters in the clustering results, the number of female and male flowers in the current inflorescence was obtained respectively.

[0067] S6. Divide the dataset and use the Bayesian optimization algorithm to tune the key hyperparameters in the D-NeRF model and PointNet++ model training process. Specifically,

[0068] To ensure the generalization ability of the model, the dataset is divided into training set, validation set and test set in a ratio of 8:1:1;

[0069] A Bayesian optimization algorithm was used to jointly tune the key hyperparameters during D-NeRF and PointNet++ model training. During D-NeRF model training, the optimized hyperparameters included the learning rate, the number of sampling points per ray (coarse and fine sampling points), the position encoding frequency, the time encoding dimension, and the number of rays per training round (raysperbatch). During PointNet++ model training, the optimized hyperparameters included the learning rate, batch size, the number of sampling points, the subsampling hierarchy (such as the number of sampling points and radius settings per layer), and the dropout probability.

[0070] The objective function of the tuning is to maximize the F1-score on the validation set. The evaluation indicators include:

[0071]

[0072] Among them, Precision and Recall represent the accuracy and recall rate respectively.

[0073] By using the Bayesian optimization method to search in the multi-dimensional hyperparameter space, the model's performance in classifying male and female flowers on the validation set was effectively improved, ensuring that the system achieved a good balance between accuracy and recall.

[0074] The method of the present invention integrates NeRF reconstruction, point cloud processing, deep learning target detection and mobile terminal deployment to achieve three-dimensional reconstruction of fruit tree inflorescences and identification and counting of male and female flowers. The method innovatively combines the NeRF model and PointNet++ to form a complete and feasible intelligent detection method for fruit tree inflorescences. The method solves the problems of traditional two-dimensional image detection being sensitive to spatial occlusion, low counting accuracy and limited information dimension. By introducing D-NeRF technology, high-quality three-dimensional point cloud reconstruction of the fruit tree inflorescence structure is achieved, and combined with a point cloud classification network based on PointNet++, accurate identification and quantity counting of male and female flowers of panicle fruit trees are achieved.

[0075] Example 1

[0076] This embodiment takes the counting of male and female flowers in litchi inflorescence as an example, and includes the following steps:

[0077] 1. Image acquisition and preprocessing, including:

[0078] An RGB camera was used to capture the litchi inflorescence in a circular pattern. During the recording process, the distance and lighting were kept constant, and wind interference was avoided. The resulting video was saved in MP4 format. FFmpeg was used to extract images frame by frame, extracting every two frames. These images were saved in PNG format and renamed and numbered for ease of subsequent manipulation.

[0079] The images were uniformly cropped and scaled to 790×480 pixels, color normalized, and enhanced using OpenCV. The COLMAP tool was then used to extract the camera pose parameters for each frame and exported to the transform.json format.

[0080] 2. 3D reconstruction based on D-NeRF, specifically including:

[0081] The image sequence, along with its corresponding camera parameters and timestamps, is fed into the D-NeRF model for training. The model consists of two multilayer perceptrons: a canonical network and a deformable network. These networks learn a static scene representation in a canonical space and a temporal deformation mapping, respectively, to achieve 3D reconstruction of dynamic scenes. The training process optimizes the model by minimizing the mean squared error between rendered and true colors.

[0082] After training is completed, the density threshold method is used to extract high-density points to generate a three-dimensional point cloud, which is then exported to PLY format for subsequent processing.

[0083] 3. Point cloud annotation and classification training, specifically including:

[0084] In CloudCompare, we denoised the 3D point cloud generated by the D-NeRF model and annotated the pistillate and staminate flowers, as well as the background. We also completed the annotation of each flower instance. The annotated point cloud was exported as .txt and .ply files containing the location, color, and category.

[0085] The PointNet++ deep neural network model is trained for male and female flower identification. The point cloud is divided into multiple 1024-point sub-blocks, normalized, and then fed into the model. The model architecture includes sampling, neighborhood construction, and local feature extraction modules. The training process uses a cross-entropy loss function and parameter optimization using the Adam optimizer. After model training is complete, male and female flower points can be classified in unknown inflorescence point cloud data.

[0086] After classification, the DBSCAN clustering algorithm is used to perform cluster statistics on the point sets of female flowers and male flowers respectively to achieve automatic counting of the number of male and female flowers.

[0087] Example 2

[0088] This embodiment provides a system for counting male and female flowers of panicle fruit trees through 3D reconstruction. The system adopts the counting method provided by the present invention and includes a front-end applet module and a cloud-based inference service module.

[0089] The front-end mini-program module uses the mobile mini-program as an interactive platform to provide a data interaction interface between users and the system. Its main functions include:

[0090] Inflorescence video collection and compression upload: Call the mobile device camera to collect inflorescence video data and perform local compression processing on the collected video to reduce the network transmission burden. The compression format can use mainstream encoding formats such as MP4 or H.264;

[0091] Recognition result display and interactive visualization: After receiving the recognition results returned by the cloud-based inference service module, the mini program supports visual display of the reconstructed 3D inflorescence model, including: category labeling of female and male flower points (distinguished by different colors or marks); display of ID numbers of different flower instances; graphical output of the statistical results of the number of flowers of each type (such as a bar chart or numerical labels); support for interactive operations such as rotation and zooming, making it easy for users to view the inflorescence structure and recognition results;

[0092] Processing progress query and status feedback: Provides a processing status query interface so that users can understand the progress of each stage such as task uploading, queuing, calculation and result generation in real time, improving user experience.

[0093] The cloud-based inference service module is deployed on the cloud server and includes a video reception and preprocessing module, a D-NeRF 3D reconstruction module, a point cloud recognition and male and female flower classification module, and a result encoding and transmission module.

[0094] The video receiving and preprocessing module uses a SpringBoot-based Web framework to build a data receiving service, which receives and parses the video uploaded by the front-end applet module; samples and extracts the video to generate an image sequence for 3D reconstruction;

[0095] The D-NeRF 3D reconstruction module calls the D-NeRF model deployed in the cloud to reconstruct the 3D point cloud structure of the inflorescence based on the acquired image sequence. This module uses GPU-accelerated inference to ensure that the model reconstruction task is completed in a reasonable time.

[0096] The point cloud recognition and male and female flower classification module segments and preprocesses the reconstructed 3D point cloud, then inputs the trained PointNet++ model to perform male and female flower recognition at each point and output the classification results.

[0097] The result encoding and transmission module compresses and encodes key information in the classification results (such as point category, instance ID, flower count, etc.), retains only necessary fields (such as XYZ coordinates + classification label), and returns them to the front-end mini-program module in a lightweight JSON format;

[0098] In the cloud-based inference service module, the trained D-NeRF and PointNet++ models are exported in ONNX format and combined with the TensorRT tool for graph structure optimization, quantization, and pruning. This further compresses the model size, improves inference efficiency, reduces cloud resource consumption, and enhances the overall system's response speed and processing capabilities.

[0099] like Figure 4 The figure shows a schematic diagram of system deployment in this embodiment.

[0100] The system in this embodiment supports two processing modes:

[0101] In online processing mode, under a stable network environment, the front-end applet module directly uploads the collected tasks to the cloud-based inference service module, completes 3D reconstruction and recognition tasks in real time, and returns recognition results within minutes;

[0102] In offline acquisition plus delayed processing mode, when the network is unstable or there is no network, the front-end applet module will cache the collected video data locally, and then upload it in batches to the cloud inference service module for processing after the device restores the network connection; the system will automatically identify offline tasks and arrange queue processing, and users can view the complete recognition results later.

[0103] This embodiment combines actual deployment needs and designs an efficient deployment architecture of "front-end applet collection + cloud-based model inference + front-end result display". It is compatible with online and offline processing processes, solving the problem that existing mobile devices are difficult to carry out high-complexity model operations. While ensuring recognition accuracy, it achieves lightweight deployment and friendly interaction of the system. It has strong scalability and practicality, and is suitable for the actual application needs of automatic monitoring of fruit tree flower quantity in the field of smart agriculture.

[0104] It should also be noted that, in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0105] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction, characterized in that: The following steps are involved: S1, collect image sequences, use a camera to shoot inflorescence video, and extract the video into images frame by frame; S2, image preprocessing and camera parameter estimation, preprocess the inflorescence images and use the COLMAP tool to obtain the camera parameters corresponding to each image; S3, training the D-NeRF model and using the D-NeRF model to perform multimodal 3D reconstruction of inflorescence point clouds; S4, optimize and annotate the 3D point cloud output by the D-NeRF model; S5. Train the PointNet++ model and use it to identify and count male and female flowers. S6. Divide the dataset and use the Bayesian optimization algorithm to tune the key hyperparameters in the training process of the D-NeRF model and PointNet++ model.

2. The method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction according to claim 1, characterized in that: Step S1 specifically includes: Use a camera to photograph the inflorescence, moving the camera around the inflorescence and shooting from different angles while maintaining a constant distance between the camera and the inflorescence. The shooting angles should cover all directions of the inflorescence, including up and down, left and right, and front and back. The inflorescence should be kept fixed during the shooting process. After shooting, the video is saved and FFmpeg is used to extract the video frame by frame into images. One image is extracted every two frames to obtain a set of image sequences in time order. Finally, the images are numbered in order.

3. The method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction according to claim 2, characterized in that: Step S2 specifically includes: Crop and scale the inflorescence image and unify the image size; perform color normalization on the image and scale the pixel values ​​to between 0 and 1; Check the image and remove blurry or unsharp images caused by jitter. Use OpenCV filtering methods to sharpen blurry or unevenly exposed images. The sharpening process includes Gaussian filtering or sharpening. Use the COLMAP tool to obtain the camera parameters corresponding to each image, that is, the camera position and orientation; input the image into COLMAP, use the Structure-from-Motion algorithm to calculate the camera's extrinsic and intrinsic parameters, output the camera pose and corresponding sparse point cloud for each image, and finally save the camera pose parameters.

4. The method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction according to claim 1, characterized in that: Step S3 specifically includes: S31, input data, including image sequence, camera parameters of each image and its corresponding timestamp; S32, training D-NeRF model; D-NeRF model consists of two fully connected neural networks MLP, the standard network Ψ x and deformable network Ψ t ; Standardize network x , learning the static implicit representation of the scene in the canonical space, outputting the volume density σ and view-dependent color c of the 3D point x; deformation network Ψ t , modeling the time domain deformation field, mapping the observation space coordinates (x, t) to the standard space coordinates x′=x+Δx, where Δx=Ψ t (x,t); The dynamic volume rendering equation calculates the color on the camera ray r(h) = o + hd as follows: Among them, x ‘ =r(h)+Ψ t (r(h,t)) is the canonical space coordinate after deformation, is the cumulative transmittance, σ and c are the volume density and color respectively; During the training process, some pixels are randomly selected from the image in each round, and the difference between the color rendered by the model and the color of the original image is calculated to continuously optimize the network weights. After the training is completed, the mean square error (MSE) between the rendered pixels and the real pixels is minimized, and the norm network Ψ is jointly optimized. x and Deformable Network Ψ t The model outputs a time-varying 3D implicit field, which can generate high-resolution images at any time and angle of view and corresponding time-space consistent point cloud data through dynamic volume rendering. S33. Render and export the 3D point cloud. The trained D-NeRF model can generate a new image from any perspective. Using the D-NeRF model, rays are sampled from multiple angles, and the density values ​​output by the model are converted into the positions of volume points. A density threshold is set, and points exceeding the density threshold are considered to be points where real objects exist. The entire 3D space is traversed, and high-density points are collected to obtain a dense point set, namely the 3D point cloud.

5. The method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction according to claim 1, characterized in that: Step S4 specifically includes: S41, point cloud denoising and enhancement, load the 3D point cloud generated by the D-NeRF model into CloudCompare software, and use the Noise filter to remove outliers; S42, annotation content, manually annotate the inflorescence area in the 3D point cloud generated by the D-NeRF model to distinguish between male and female flowers and the background; Use hierarchical annotation tools to annotate point clouds, including: Semantic annotation: mark the female flowers, male flowers, and background in the point cloud. Each point contains (x, y, z, label), label∈{female flower, male flower, background}; Instance labeling: assigning a unique ID to each flower instance to support counting tasks; S43. Export the annotation data. Select the point cloud object in the CloudCompare software and export it into a txt file. The file contains the coordinates, color, and label value of each point.

6. The method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction according to claim 1, characterized in that: Step S5 specifically includes: S51. Preprocessing the original point cloud data; dividing the three-dimensional point cloud of the entire inflorescence into multiple local sub-point cloud blocks using a spatial partitioning method, each sub-point cloud block containing a fixed number of points; if the number of points in a sub-block is insufficient, repeated sampling is used to fill it; if the number of points exceeds, random downsampling is used to intercept it; normalizing the coordinates of all points; each sub-point cloud block retains the original category label information; S52. Train the PointNet++ model. The PointNet++ model consists of multiple layers of feature extraction modules, each of which includes point sampling, neighborhood query, and local feature extraction operations. The model input is the point coordinate information of each sub-point cloud block, and the output is the category probability of each point. During training, the cross-entropy loss function is used for error backpropagation. The Adam optimizer is used, the learning rate is set to 0.001, and the number of iterations is determined based on the accuracy of the validation set. S53. After training, the PointNet++ model is used to predict the reconstructed inflorescence 3D point cloud data. Specifically: First, the complete point cloud is divided into several sub-point cloud blocks according to the same block division method as in training. These blocks are then input into the trained PointNet++ model to obtain the classification probability of each point. Finally, a predicted label is assigned to each point to achieve the classification of male and female flower points. S54, counting the male and female flowers based on the male and female flower classification results, specifically: The point clouds classified as female flowers and male flowers were extracted into two independent subsets respectively. The density-based spatial clustering algorithm DBSCAN was applied to the two sub-point cloud sets. The algorithm does not require a preset number of clusters and can identify the distribution of point clusters in natural growth structures. By counting the number of clusters in the clustering results, the number of female and male flowers in the current inflorescence was obtained respectively.

7. The method for counting male and female flowers of panicle fruit trees by three-dimensional reconstruction according to claim 1, characterized in that: Step S6 specifically includes: To ensure the generalization ability of the model, the dataset is divided into training set, validation set and test set in a ratio of 8:1:1; A Bayesian optimization algorithm was used to jointly tune key hyperparameters during D-NeRF and PointNet++ model training. During D-NeRF model training, the optimized hyperparameters included the learning rate, number of sampling points per ray, position encoding frequency, time encoding dimension, and number of rays per training round. During PointNet++ model training, the optimized hyperparameters included the learning rate, batch size, number of sampling points, subsampling hierarchy, and dropout probability. The objective function of the tuning is to maximize the F1-score on the validation set. The evaluation indicators include: Among them, Precision and Recall represent the accuracy and recall rate respectively.

8. A panicle-type fruit tree flower 3D reconstruction and counting system for male and female flowers, characterized in that: The system adopts the method according to any one of claims 1 to 7, and the system includes a front-end applet module and a cloud-based reasoning service module; The front-end applet module is used for inflorescence video collection, compression and upload, recognition result display and visual interaction, processing progress query and status feedback; The cloud-based inference service module is deployed on the cloud server and includes a video reception and preprocessing module, a D-NeRF 3D reconstruction module, a point cloud recognition and male and female flower classification module, and a result encoding and transmission module.

9. The panicle fruit tree flower 3D reconstruction and male and female flower counting system according to claim 8, characterized in that: The video reception and preprocessing module uses a SpringBoot-based Web framework to build a data reception service, receive and parse videos uploaded by the front-end applet module; sample and extract the video to generate an image sequence for 3D reconstruction; The D-NeRF 3D reconstruction module calls the D-NeRF model deployed in the cloud to reconstruct the 3D point cloud structure of the inflorescence based on the collected image sequence; The point cloud recognition and male and female flower classification module segments and preprocesses the reconstructed 3D point cloud, then inputs the trained PointNet++ model to perform male and female flower recognition at each point and output the classification results. The result encoding and transmission module compresses and encodes the key information in the classification results, retaining only the necessary fields, and returns them to the front-end mini-program module in a lightweight JSON format; In the cloud-based inference service module, the trained D-NeRF and PointNet++ models are exported in ONNX format and combined with the TensorRT tool for graph structure optimization, quantization, and pruning. This further compresses the model size, improves inference efficiency, reduces cloud resource consumption, and enhances the overall system's response speed and processing capabilities.

10. The panicle fruit tree flower 3D reconstruction and male and female flower counting system according to claim 9, characterized in that: The system supports two processing modes: In online processing mode, under a stable network environment, the front-end applet module directly uploads the collected tasks to the cloud-based inference service module, completes the 3D reconstruction and recognition tasks in real time, and then returns the recognition results; In offline acquisition plus delayed processing mode, when the network is unstable or there is no network, the front-end applet module will cache the collected video data locally, and then upload it in batches to the cloud inference service module for processing after the device restores the network connection; the system will automatically identify offline tasks and arrange queue processing, and users can view the complete recognition results later.

Images