Three-dimensional modeling method and system of plant leaves based on three-dimensional network database

Abstract

The present invention discloses a three-dimensional modeling method and system for plant leaves based on a three-dimensional network database, including: using an improved VMamba model to accurately identify from an input two-dimensional picture; based on the above classification information and a rich three-dimensional model library of leaves constructed, finding a mesh model of the required deformation; using an improved LiteHRNet model to find the key points of the main veins of the mesh model, and using a SAM model to segment the contour features to obtain the key points of the contour features of the mesh model; comparing the main veins and contour features, deforming the mesh model, and then generating a three-dimensional mesh model of the leaf. The present invention significantly enhances the ability of a single image to capture and express plant geometric properties by accurately extracting the main veins and contour features of the leaf; not only can it accurately perform three-dimensional modeling, but it also effectively solves the problems caused by multi-view images in traditional image-based three-dimensional reconstruction technology, and significantly improves user experience.

Description

Technical Field

[0001] The present invention relates to a three-dimensional reconstruction technology in computer vision technology, and more specifically, to a three-dimensional modeling method and system for plant leaves based on a three-dimensional network database. Background Art

[0002] Image-based 3D reconstruction technology is a method of generating a 3D model of an object or scene by processing 2D image data. This technology is widely used in computer vision, virtual reality, cultural heritage protection and other fields. Usually, image-based 3D reconstruction relies on images taken from multiple perspectives to build a 3D model through feature matching, disparity calculation and depth information extraction. Common methods include stereo vision (Stereo Vision), Multi-View Stereo (MVS) and photogrammetry (Photogrammetry).

[0003] Although image-based 3D reconstruction technology has made significant progress in accuracy and wide application, it still faces many challenges in practical applications. High-quality 3D reconstruction usually requires a large number of images taken from multiple different perspectives. This is very difficult in dynamic scenes or constrained environments. In addition, the acquisition process of multi-view images is complex and time-consuming, and processing and matching a large number of images requires powerful computing resources, especially in high-resolution or large-scale scenes. Large-scale data processing not only requires high-performance hardware support, but also causes long computing delays, affecting real-time performance. In actual scenes, images may be affected by factors such as noise, occlusion, and illumination changes, making it difficult to extract and match feature points, affecting the accuracy and integrity of the reconstructed model. In addition, the deployment and calibration process of multi-camera systems is complex and easily affected by errors. Problems such as sensor fusion and synchronization also further increase the complexity and cost of the system, which is not conducive to widespread application. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a three-dimensional modeling method and system for plant leaves based on a three-dimensional network database. This solution simplifies data acquisition, reduces computing resource requirements, reduces the complexity of the reconstruction system, and provides an efficient, convenient and more accurate leaf three-dimensional modeling solution.

[0005] A first aspect of the present invention provides a three-dimensional modeling method for plant leaves based on a three-dimensional network database, comprising:

[0006] Collect leaves of various types and at different times, scan them with a 3D scanner, and build a 3D network database of multiple types of leaves;

[0007] Obtaining classification information of an input blade image, and finding a mesh model of a required deformation in a constructed three-dimensional network database of multiple types of blades based on the classification information;

[0008] Generate main vein feature information according to the input leaf image, and generate a main vein trend function based on the main vein feature information;

[0009] Generate blade edge contour feature information according to the input blade image, and generate an edge contour point set based on the blade edge contour feature information;

[0010] The network model is translated and rotated to a corresponding position, and points on the main veins of the network model are projected onto the maximum imaging plane to obtain an initialized network model;

[0011] The main vein trend function of the input leaf is used to deform the main vein of the network model, and the edge contour points of the grid model are mapped with the edge contour point set of the input leaf to generate a three-dimensional model of the input leaf.

[0012] In this scheme, leaves of various types and at different times are collected, and the leaves are scanned by a 3D scanner to build a 3D network database of multiple types of leaves, specifically:

[0013] Use a 3D scanner to scan the blade, obtain and process the geometric shape and appearance information of the blade, fuse the pre-processed geometric shape and appearance information, and realize the alignment and merging of the scanned data;

[0014] A blade 3D model is created based on the processed scan data, a label of the blade 3D model is set using the blade type information, a multi-category blade 3D network database is constructed, and the labeled blade 3D model is added to the multi-category blade 3D network database.

[0015] In this solution, the classification information of the input blade image is obtained, and the mesh model of the required deformation is retrieved from the blade three-dimensional network database based on the classification information, specifically:

[0016] Obtaining a multivariate sequence containing input leaf classification information, obtaining the classification information using an improved VMamba model, encoding the multivariate sequence through an Embedding encoding block, and obtaining an initial feature representation;

[0017] The initial feature representation is passed to two VSS feature extraction blocks, the output of each VSS feature extraction block is introduced into a linear embedding layer for embedding representation, the output of the linear embedding layer is divided into two information streams, the outputs of the two information streams are multiplied to generate the final output of the VSS feature extraction block;

[0018] The features of the final output of the VSS feature extraction block are fused through the Merging block, and the features output by the Merging block are passed to two other VSS feature extraction blocks for further feature extraction;

[0019] After passing through the Merging block again, the feature fusion process of the upper layer is repeated, and the features output by the Merging block are passed to the nine VSS feature extraction blocks for deeper feature extraction;

[0020] The acquired deep-level features are subjected to the final feature fusion processing by the Merging block. The features output by the Merging block are passed to two VSS feature extraction blocks to complete the final feature extraction and output the classification information of the blade types. Based on the classification information, the mesh model with the required deformation is found in the constructed three-dimensional network database of multiple types of blades.

[0021] In this solution, the main vein feature information is generated according to the input leaf image, and the main vein trend function is generated based on the main vein feature information, specifically:

[0022] The image of the input leaf is obtained, and the improved LiteHRNet model is used to locate the main vein points. The improved LiteHRNet model is divided into four stages;

[0023] In the first stage, the convolutional layer is used to reduce the resolution of the image to the resolution of the heat map, which is 1 / 4 of the original image resolution. At the same time, the number of channels is increased to 32, and a 1 / 2 resolution branch is generated using the first transition layer;

[0024] In the second stage, different numbers of VSS modules are stacked according to the number of layers of the branch to learn and extract spatial information. After feature extraction, the information of each layer is fused through the fusion layer, and a branch with 1 / 4 resolution is generated using the second transition layer.

[0025] In the third stage, different numbers of VSS modules are stacked according to the number of layers of the branch to learn and extract spatial information. After feature extraction, the information of each layer is fused through the fusion layer, and a branch with a resolution of 1 / 8 is generated using the third transition layer.

[0026] In the fourth stage, the low-resolution branches in the first three stages are upsampled to make their resolutions consistent with the initial thermal Figure 1 Through a 1×1 convolutional layer with 11 output channels, a heat map that accurately corresponds to the key points of the main veins of the input leaves is obtained as the final output result, and the main vein trend function is generated according to the key points of the main veins.

[0027] In this solution, the blade edge contour feature information is generated according to the input blade image, and the edge contour point set is generated based on the blade edge contour feature information, specifically:

[0028] The pre-trained SAM model is used to extract the edge contour feature information of the leaf in the input leaf image, and the features are adjusted through the adaptation layer to adapt to the leaf segmentation task;

[0029] Get the feature map of the leaf image, apply an upsampling layer on the extracted feature map to restore the feature map to the original resolution, use a convolution layer on each restored feature map, output the heat map of the petiole point and the leaf tip point, and normalize it through the Softmax function;

[0030] Find the location of the maximum value on each heat map as the coordinates of the predicted key points, and refine it through Gaussian peak adjustment to accurately locate the petiole point and leaf tip point as the key points;

[0031] The complete outline of the leaf is generated by combining the key points and feature maps. The improved LiteHRNet model used in extracting the main vein feature information is called to locate the contour key points and generate an edge contour point set.

[0032] In this solution, the network model is translated and rotated to the corresponding position, and the points on the main veins of the network model are projected onto the maximum imaging plane, specifically:

[0033] Get the coordinate position (x b ,y b ,z b ), translate the mesh model to translate the blade tip to the origin, and obtain the coordinate position of the blade tip of the mesh model (x j ,y j ,z j ), multiply the mesh model by the rotation matrix to rotate the blade tip to the x-axis;

[0034] Use CGAL to traverse each convex hull surface in the network model, calculate the normal vector of each face by obtaining the half edge and the corresponding vertex on the face, store it as a normal vector array, traverse the normal vector array in a loop, calculate the sum of all normal vectors as the geometric center, divide the geometric center by the number of normal vectors, get the average value, and get the geometric center of the normal vector;

[0035] Calculate the deviation between each normal vector and the geometric center, add the outer accumulation of the deviation to the covariance matrix, and then divide it by the number of normal vectors to get the average value, solve the eigenvalue and eigenvector of the covariance matrix, and take the eigenvector corresponding to the maximum eigenvalue as the normal vector γ in the direction of the maximum projection surface;

[0036] Find a vector α on the plane perpendicular to the normal vector γ, and multiply the vector α with the normal vector γ to get the vector β, and establish a rectangular coordinate system;

[0037] The points of the main veins in the grid model are denoted as (x m ,y m , z m ), according to the formula (x m ,y m , z m )(α T , β T ) Project the main vein point onto the maximum imaging plane to obtain the coordinates (x1, y1) of the main vein point in the grid model in the maximum projection plane.

[0038] In this scheme, the main vein trend function of the input leaf is used to deform the main vein of the network model, and the edge contour points of the grid model are mapped to the edge contour point set of the input leaf, specifically:

[0039] Obtain the coordinates (x1, y1) of the main vein point in the grid model in the maximum projection plane, use the least squares method to fit the main vein trend function that is the same as the main vein trend of the network model, substitute the horizontal coordinate x1, calculate the function value y2, and then calculate the distance d1 between y2 and y1;

[0040] The least square method is used to fit the main vein trend function of the input leaf, the horizontal coordinate x1 is substituted into the function value y3, and then the distance d1 is added to the function value y3 to obtain the vertical coordinate y4;

[0041] Subtract the ordinate y1 from the ordinate y4 to get the required translation distance d2. According to the formula d2β+(x m ,y m , z m ), β is the vector in the maximum projection plane, and the point of the main vein in the network model after translation is calculated as (x1, y1, z1);

[0042] After the key points of the network model are translated, the size and alignment of the input leaf image are adjusted to ensure that the endpoints of the input leaf correspond to the endpoints of the mesh model. According to the shape of the edge contour points in the input leaf image, the translated mesh model is deformed accordingly to generate a three-dimensional model of the input leaf.

[0043] The second aspect of this aspect provides a plant leaf three-dimensional modeling system based on a three-dimensional network database, including: a database construction module, a leaf prediction module, a local feature extraction module, a global feature extraction module, a model initialization module, and a leaf three-dimensional deformation module:

[0044] The database construction module collects leaves of various types and at different times, and scans the leaves through a 3D scanner to construct a three-dimensional network database of multiple types of leaves;

[0045] The blade prediction module is used to obtain the classification information of the input blade, and find the grid model of the required deformation in the three-dimensional network database of multiple types of blades based on the classification information;

[0046] The local feature extraction module generates main vein feature information according to the input leaf image, and generates a main vein trend function based on the main vein feature information;

[0047] The global contour feature extraction module extracts the feature information of the blade edge contour according to the input blade image, and generates a point set of the edge contour based on the feature information of the blade edge contour;

[0048] The model initialization module translates the network model based on the network model selected in the leaf prediction module so that the petiole point is located at the origin, rotates the model so that the leaf tip point falls on the x-axis, and then projects the points on the main veins in the network model onto the maximum imaging plane;

[0049] The leaf three-dimensional deformation module is based on the grid model initialized in the model initialization module, and deforms the main veins of the network model according to the main vein trend function of the input leaf, and maps the edge contour points of the grid model with the edge contour point set of the input leaf to obtain the three-dimensional model of the input leaf.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) Classify the input leaves and match them according to the classification information to make the found mesh model more accurate. This operation can facilitate subsequent deformation and also provide a basis for the similarity of the target mesh model obtained by deformation.

[0052] (2) Extracting high-level semantic category attributes of the input leaf, including main vein information and edge contour information, can restore the leaf model more realistically.

[0053] (3) Since the input blade only provides two-dimensional information, when the mesh model is deformed, it is performed in the direction of the maximum projection surface instead of simply discarding the z-axis to avoid distortion of the reconstructed blade model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the drawings required for use in the embodiments or exemplary descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to the drawings without paying creative work.

[0055] Figure 1 A flow chart of a method for three-dimensional modeling of plant leaves based on a three-dimensional network database according to an embodiment of the present invention is shown;

[0056] Figure 2 The network structure diagram of the improved VMamba model of the present invention is shown;

[0057] Figure 3 The network structure diagram of the improved LiteHRNet model of the present invention is shown;

[0058] Figure 4 A block diagram of a plant leaf three-dimensional modeling system based on a three-dimensional network database according to an embodiment of the present invention is shown; DETAILED DESCRIPTION

[0059] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0060] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0061] Figure 1 A flow chart of a method for three-dimensional modeling of plant leaves based on a three-dimensional network database according to an embodiment of the present invention is shown.

[0062] like Figure 1 As shown, the present invention provides a three-dimensional modeling method of plant leaves based on a three-dimensional network database, comprising:

[0063] S102, collecting leaves of various types and at different times, scanning the leaves with a 3D scanner, and constructing a three-dimensional network database of multiple types of leaves;

[0064] S104, obtaining classification information of the input blade image, and finding a mesh model of required deformation in a constructed three-dimensional network database of multiple types of blades based on the classification information;

[0065] S106, generating main vein feature information according to the input leaf image, and generating a main vein trend function based on the main vein feature information;

[0066] S108, generating blade edge contour feature information according to the input blade image, and generating an edge contour point set based on the blade edge contour feature information;

[0067] S110, translating and rotating the network model to a corresponding position, and projecting points on the main veins of the network model onto a maximum imaging plane to obtain an initialized network model;

[0068] S112, using the main vein trend function of the input leaf to deform the main vein of the network model, and mapping the edge contour points of the grid model with the edge contour point set of the input leaf to generate a three-dimensional model of the input leaf.

[0069] It should be noted that when a 3D scanner is used to scan the blade, the scanning device will emit laser or structured light. Based on the received light signal, the computer can determine information such as the shape and texture of the object, including operations such as removing noise and filling holes; obtain and process the geometric shape and appearance information of the blade, and fuse the pre-processed geometric shape and appearance information to achieve alignment and merging of scanning data from different perspectives; create a three-dimensional model of the blade based on the processed scanning data, and refine, modify and optimize it, use the blade type information to set the label of the blade three-dimensional model, build a three-dimensional network database of multiple types of blades, and add the labeled three-dimensional model of the blade to the three-dimensional network database of multiple types of blades.

[0070] Figure 2 The network structure diagram of the improved VMamba model of the present invention is shown;

[0071] It should be noted that the multivariate sequence containing the input leaf classification information is obtained, and the classification information is obtained using the improved VMamba model. The improved VMamba model is divided into four layers of networks, specifically:

[0072] In the first layer of the network, the multivariate sequence is encoded by the Embedding encoding block to obtain the initial feature representation;

[0073] The initial feature representation is passed to two VSS feature extraction blocks, and the output of each VSS feature extraction block is imported into the linear embedding layer for embedding representation. The output of the linear embedding layer is divided into two information flows: Information flow 1: passes through a 3x3 depth-separable convolution layer, then passes through the SiLU activation function, and then enters the core SS2D module. The output of the SS2D module is processed by a layer normalization layer; Information flow 2: directly multiply the outputs of the two information flows through a SiLU activation function to generate the final output of the VSS feature extraction block;

[0074] In the second layer of the network, the features of the final output of the VSS feature extraction block are fused through the Merging block: Upper path: The features pass through a 2x2 pooling layer with a stride of 2. Lower path: The features first pass through a 1x1 convolution to increase the number of channels, then pass through a 4x4 depthwise separable convolution layer (DW convolution) with a stride of 2, and then restore the number of channels through a 1x1 convolution. The two outputs are spliced ​​along the channel dimension, and finally a 1x1 convolution is performed for channel fusion. The features output by the Merging block are passed to two other VSS feature extraction blocks for further feature extraction;

[0075] In the third layer of the network, the Merging block is passed again to repeat the feature fusion process of the upper layer. The features output by the Merging block are passed to nine VSS feature extraction blocks for deeper feature extraction.

[0076] In the fourth layer of the network, the acquired deep-level features are subjected to the final feature fusion processing by the Merging block. The features output by the Merging block are passed to two VSS feature extraction blocks to complete the final feature extraction and output the classification information of the blade types. Based on the classification information, the mesh model with the required deformation is found in the constructed three-dimensional network database of multiple types of blades.

[0077] The improved VMamba model is used in the embodiment of the present invention. The model includes four network layers. The first network layer includes an Embedding encoding block and two VSS feature extraction blocks, the second network layer includes a Merging block and two VSS feature extraction blocks, the third network layer includes a Merging block and nine VSS feature extraction blocks, and the fourth network layer includes a Merging block and two VSS feature extraction blocks. Finally, the model output is obtained through a classification head including a global pooling layer and a fully connected layer. Each time the Merging block is passed, the height and width of the feature map will be reduced by half, and the number of channels will be doubled.

[0078] The selective scanning mechanism in SS2D in VSS Block refers to B∈R in state transfer B×L×N , C∈R B×L×N ,Δ∈R B×L×D , all come from the input data x∈R B×L×D , indicating that the state-space model can utilize the contextual information in the input data to ensure that the weights in the mechanism are dynamic. Using the four-way 2D selective scanning mechanism, after the scanning is completed, the results of the four-way scanning are serialized, and then the state-space model is used for selective scanning, and finally the fusion is restored.

[0079] To alleviate the problem of small receptive fields of Patch Embedding and Patch Merging in VMamba, a Merging block with the feature of fusing features of different scales is constructed here. This block combines separable depthwise convolution, point convolution, and average pooling to fuse feature maps, while reducing the height and width of feature maps and increasing the number of feature map channels. This design helps the network extract more abstract high-level features from input features, allowing the network to maintain efficient information processing capabilities while reducing calculations and parameters, while also increasing the receptive field of the model.

[0080] Figure 3 A flow chart of the leaf vein key point positioning model method of the present invention is shown.

[0081] It should be noted that the main vein feature information is generated according to the input leaf image, and the main vein trend function is generated based on the main vein feature information. The image of the input leaf is obtained, and the improved LiteHRNet model is used to locate the main vein point. The improved LiteHRNet model is divided into four stages; each stage (stage) is composed of multiple branches, and these branches achieve deep extraction of image features by stacking VSS Block modules of different numbers and scales. Between the stages, the transition layer (Transition Layer) and the fusion layer (Fuse Layer) Fuse Layer are cleverly used to connect to ensure that information can be smoothly transmitted and fused between different branches. In the final output link, the fourth stage (stage4) fuses the feature information of each branch through a Fuse Layer, so that feature maps of different resolutions can complement each other to form a more comprehensive and accurate feature representation. This process not only retains the spatial information at high resolution, but also incorporates more abstract semantic information, thereby greatly improving the accuracy of the model in locating the key points of the main veins of the leaf.

[0082] In the first stage (stage 1), the Conv1 module uses two 3×3 convolutional layers to reduce the image resolution to the resolution of the heat map, which is 1 / 4 of the original image resolution. At the same time, the number of channels is increased to 32, and the first transition layer (Transition Layer1) is used to generate a branch with 1 / 2 resolution;

[0083] In the second stage (stage2), different numbers of VSS modules are stacked according to the number of layers of the branch to learn and extract spatial information. After feature extraction, the information of each layer is fused through the fuse operation to ensure that the low-resolution layer can learn spatial information and the high-resolution layer can also obtain more abstract semantic features. Use the second transition layer (Transition Layer2) to generate a 1 / 4 resolution branch;

[0084] In the third stage (stage3), different numbers of VSS modules are stacked according to the number of layers of the branch to learn and extract spatial information. After feature extraction, the information of each layer is fused through the fusion layer, and a 1 / 8 resolution branch is generated using the third transition layer (TransitionLayer3);

[0085] In the fourth stage (stage 4), the low-resolution branches in the first three stages are upsampled to make their resolutions consistent with the initial hotspot. Figure 1 Through a 1×1 convolutional layer (Conv2d module) with 11 output channels, a heat map that accurately corresponds to the key points of the main veins of the input leaves is obtained as the final output result, and the main vein trend function is generated according to the key points of the main veins.

[0086] It should be noted that the edge contour feature information of the leaf is generated according to the input leaf image, and the edge contour point set is generated based on the leaf edge contour feature information. The pre-trained SAM model is selected to extract the edge contour feature information of the leaf in the input leaf image, and the features are adjusted through the adaptation layer to adapt to the leaf segmentation task; the feature map of the leaf image is obtained, and the upsampling layer is applied to the extracted feature map to restore the feature map to the original resolution. A convolution layer is used on each restored feature map to output the heat map of the petiole point and the leaf tip point, and normalized by the Softmax function; the maximum value position is found on each heat map as the predicted key point coordinates, and refined by Gaussian peak adjustment to accurately locate the petiole point and the leaf tip point as the key point; the key points and feature map are combined to generate the complete outline of the leaf, and the improved LiteHRNet model used in the extraction of the main vein feature information is called to locate the contour key points to generate the edge contour point set.

[0087] It should be noted that the network model is translated and rotated to the corresponding position, and the points on the main veins of the network model are projected onto the maximum imaging plane: the coordinate position of the petiole point of the grid model (x b ,y b , z b ), translate the mesh model to translate the blade tip to the origin, and obtain the coordinate position of the blade tip of the mesh model (x j ,y j , z j ), multiply the mesh model by the rotation matrix to rotate the blade tip to the x-axis;

[0088] Use CGAL to traverse each convex hull surface in the network model, calculate the normal vector of each face by obtaining the half edge and the corresponding vertex on the face, store it as a normal vector array, and traverse the normal vector array in a loop to calculate the sum of all normal vectors as the geometric center, divide the geometric center by the number of normal vectors to get the average value, and get the geometric center of the normal vector; calculate the deviation of each normal vector from the geometric center, and add the outer accumulation of the deviation to the covariance matrix, and then divide it by the number of normal vectors to get the average value, solve the eigenvalue and eigenvector of the covariance matrix, and take the eigenvector corresponding to the maximum eigenvalue as the normal vector γ in the direction of the maximum projection surface; find a horizontal axis vector α on the plane perpendicular to the normal vector γ, cross-multiply the vector α with the normal vector γ to get the vertical axis vector β, and establish a rectangular coordinate system; record the point of the main vein in the grid model as (x m ,y m , z m ), according to the formula (x m ,y m , z m )(α T , β T ), T is the transpose, the main vein point is projected onto the maximum imaging plane, and the coordinates (x1, y1) of the main vein point in the grid model in the maximum projection plane are obtained.

[0089] It should be noted that the main vein trend function of the input leaf is used to deform the main vein of the network model, and the edge contour points of the mesh model are mapped to the edge contour point set of the input leaf:

[0090] Obtain the coordinates (x1, y1) of the main vein point in the grid model in the maximum projection plane, use the least squares method to fit the main vein trend function that is the same as the main vein trend of the network model, substitute the horizontal coordinate x1, calculate the function value y2, and then calculate the distance d1 between y2 and y1; use the least squares method to fit the main vein trend function of the input leaf, substitute the horizontal coordinate x1, calculate the function value y3, and then add the distance d1 to the function value y3 to obtain the vertical coordinate y4;

[0091] Subtract the ordinate y1 from the ordinate y4 to get the required translation distance d2. According to the formula d2β+(x m ,y m , z m ), β is the longitudinal axis vector in the maximum projection plane, and the point of the main vein in the network model after translation is calculated as (x1, y1, z1); considering that the fitted function curve has no width, but the main vein of the leaf model has width. In order to retain the width of the main vein of the leaf model, the distance d1 between y2 and y1 is calculated. Formula d2β+(x m ,y m , z m) is to perform a back-projection operation on the feature points originally projected on the maximum imaging plane, rather than simply discarding the z-axis, to avoid distortion of the blade model.

[0092] After the key points of the network model are translated, the size and alignment of the input leaf image are adjusted to ensure that the endpoints of the input leaf correspond to the endpoints of the mesh model. According to the shape of the edge contour points in the input leaf image, the translated mesh model is deformed accordingly to generate a three-dimensional model of the input leaf.

[0093] Figure 4 A block diagram of a plant leaf three-dimensional modeling system based on a three-dimensional network database according to an embodiment of the present invention is shown.

[0094] The second aspect of this aspect provides a plant leaf three-dimensional modeling system based on a three-dimensional network database, including: a database construction module, a leaf prediction module, a local feature extraction module, a global feature extraction module, a model initialization module, and a leaf three-dimensional deformation module:

[0095] The database construction module collects leaves of various types and at different times, and scans the leaves through a 3D scanner to construct a three-dimensional network database of multiple types of leaves;

[0096] The blade prediction module is used to obtain the classification information of the input blade, and find the grid model of the required deformation in the three-dimensional network database of multiple types of blades based on the classification information;

[0097] The local feature extraction module generates main vein feature information according to the input leaf image, and generates a main vein trend function based on the main vein feature information;

[0098] The global contour feature extraction module extracts the feature information of the blade edge contour according to the input blade image, and generates a point set of the edge contour based on the feature information of the blade edge contour;

[0099] The model initialization module translates the network model based on the network model selected in the leaf prediction module so that the petiole point is located at the origin, rotates the model so that the leaf tip point falls on the x-axis, and then projects the points on the main veins in the network model onto the maximum imaging plane;

[0100] The leaf three-dimensional deformation module is based on the grid model initialized in the model initialization module, and deforms the main veins of the network model according to the main vein trend function of the input leaf, and maps the edge contour points of the grid model with the edge contour point set of the input leaf to obtain the three-dimensional model of the input leaf.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0102] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0103] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0104] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0105] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0106] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A three-dimensional modeling method for plant leaves based on a three-dimensional network database, characterized in that: The following steps are involved: Collect leaves of various types and at different times, scan them with a 3D scanner, and build a 3D network database of multiple types of leaves; Obtaining classification information of an input blade image, and finding a mesh model of a required deformation in a constructed three-dimensional network database of multiple types of blades based on the classification information; Generate main vein feature information according to the input leaf image, and generate a main vein trend function based on the main vein feature information; Generate blade edge contour feature information according to the input blade image, and generate an edge contour point set based on the blade edge contour feature information; The grid model is translated and rotated to a corresponding position, and points on the main veins of the grid model are projected onto a maximum imaging plane to obtain an initialized grid model; The main vein trend function of the input leaf is used to deform the main vein of the mesh model, and the edge contour points of the mesh model are mapped with the edge contour point set of the input leaf to generate a three-dimensional model of the input leaf; The classification information of the input blade image is obtained, and based on the classification information, a mesh model of the required deformation is retrieved from the blade three-dimensional network database, specifically: Obtaining a multivariate sequence containing input leaf classification information, obtaining the classification information using an improved VMamba model, encoding the multivariate sequence through an Embedding encoding block, and obtaining an initial feature representation; The initial feature representation is passed to two VSS feature extraction blocks, the output of each VSS feature extraction block is introduced into a linear embedding layer for embedding representation, the output of the linear embedding layer is divided into two information streams, the outputs of the two information streams are multiplied to generate the final output of the VSS feature extraction block; The features of the final output of the VSS feature extraction block are fused through the Merging block, and the features output by the Merging block are passed to two other VSS feature extraction blocks for further feature extraction; After passing through the Merging block again, the feature fusion process of the upper layer is repeated, and the features output by the Merging block are passed to the nine VSS feature extraction blocks for deeper feature extraction; The acquired deep-level features are subjected to the final feature fusion processing by the Merging block. The features output by the Merging block are passed to two VSS feature extraction blocks to complete the final feature extraction and output the classification information of the blade types. Based on the classification information, the mesh model with the required deformation is found in the constructed three-dimensional network database of multiple types of blades.

2. A three-dimensional modeling method for plant leaves based on a three-dimensional network database according to claim 1, characterized in that: Collect leaves of various types and at different times, scan them with a 3D scanner, and build a 3D network database of multiple types of leaves, specifically: Use a 3D scanner to scan the blade, obtain and process the geometric shape and appearance information of the blade, fuse the pre-processed geometric shape and appearance information, and realize the alignment and merging of the scanned data; A blade 3D model is created based on the processed scan data, a label of the blade 3D model is set using the blade type information, a multi-category blade 3D network database is constructed, and the labeled blade 3D model is added to the multi-category blade 3D network database.

3. The method for three-dimensional modeling of plant leaves based on a three-dimensional network database according to claim 1, characterized in that: Generate main vein feature information according to the input leaf image, and generate a main vein trend function based on the main vein feature information, specifically: The image of the input leaf is obtained, and the improved LiteHRNet model is used to locate the main vein points. The improved LiteHRNet model is divided into four stages; In the first stage, the convolutional layer is used to reduce the resolution of the image to the resolution of the heat map, which is 1 / 4 of the original image resolution. At the same time, the number of channels is increased to 32, and a 1 / 2 resolution branch is generated using the first transition layer; In the second stage, different numbers of VSS modules are stacked according to the number of layers of the branch to learn and extract spatial information. After feature extraction, the information of each layer is fused through the fusion layer, and a branch with 1 / 4 resolution is generated using the second transition layer. In the third stage, different numbers of VSS modules are stacked according to the number of layers of the branch to learn and extract spatial information. After feature extraction, the information of each layer is fused through the fusion layer, and a branch with a resolution of 1 / 8 is generated using the third transition layer. In the fourth stage, the low-resolution branches in the first three stages are upsampled to make their resolution consistent with the initial heat map. Through a 1×1 convolution layer with an output channel number of 11, a heat map that accurately corresponds to the key points of the main veins of the input leaves is obtained as the final output result, and the main vein trend function is generated based on the main vein key points.

4. The method for three-dimensional modeling of plant leaves based on a three-dimensional network database according to claim 1, characterized in that: Generate blade edge contour feature information according to the input blade image, and generate edge contour point set based on the blade edge contour feature information, specifically: The pre-trained SAM model is used to extract the edge contour feature information of the leaf in the input leaf image, and the features are adjusted through the adaptation layer to adapt to the leaf segmentation task; Get the feature map of the leaf image, apply an upsampling layer on the extracted feature map to restore the feature map to the original resolution, use a convolution layer on each restored feature map, output the heat map of the petiole point and the leaf tip point, and normalize it through the Softmax function; Find the location of the maximum value on each heat map as the coordinates of the predicted key points, and refine it through Gaussian peak adjustment to accurately locate the petiole point and leaf tip point as key points; The complete outline of the leaf is generated by combining the key points and feature maps. The improved LiteHRNet model used in extracting the main vein feature information is called to locate the contour key points and generate an edge contour point set.

5. The method for three-dimensional modeling of plant leaves based on a three-dimensional network database according to claim 1, characterized in that: The grid model is translated and rotated to the corresponding position, and the points on the main veins of the grid model are projected onto the maximum imaging plane, specifically: Get the coordinate position of the petiole point of the mesh model , translate the mesh model to translate the blade tip to the origin, and obtain the coordinate position of the blade tip of the mesh model , multiply the mesh model by the rotation matrix to rotate the blade tip to the x-axis; Use CGAL to traverse each convex hull surface in the mesh model, calculate the normal vector of each face by obtaining the half edge and the corresponding vertex on the face, store it as a normal vector array, traverse the normal vector array in a loop, calculate the sum of all normal vectors as the geometric center, divide the geometric center by the number of normal vectors, get the average value, and get the geometric center of the normal vector; Calculate the deviation of each normal vector from the geometric center, add the outer accumulation of the deviation to the covariance matrix, divide it by the number of normal vectors to get the average value, solve the eigenvalue and eigenvector of the covariance matrix, and take the eigenvector corresponding to the maximum eigenvalue as the normal vector in the direction of the maximum projection surface. ; In the normal vector Find a vector on the plane perpendicular to , so that the vector With normal vector Cross product to get vector , establish a rectangular coordinate system; The points of the main veins in the grid model are denoted as , according to the formula Project the main leaf vein point onto the maximum imaging plane to obtain the coordinates of the main leaf vein point in the grid model in the maximum projection plane .

6. The method for three-dimensional modeling of plant leaves based on a three-dimensional network database according to claim 5, characterized in that: The main vein trend function of the input leaf is used to deform the main vein of the mesh model, and the edge contour points of the mesh model are mapped to the edge contour point set of the input leaf, specifically: Get the coordinates of the main vein point in the mesh model in the maximum projection surface , the least squares method is used to fit the main vein trend function that is the same as the main vein trend of the grid model, and the horizontal coordinate Substitute in and find the function value , then calculate and The distance between ; The least square method is used to fit the main vein trend function that is the same as the input leaf main vein trend. Substitute into the function and find the value , then the distance Add to function value On, get the vertical coordinate ; Using the vertical coordinate Subtract the vertical coordinate Get the required translation distance , according to the formula , is the vector in the maximum projection plane, and the point of the main vein in the translated mesh model is calculated as ; After the key points of the mesh model are translated, the size and alignment of the input leaf image are adjusted to ensure that the input leaf endpoints correspond to the endpoints of the mesh model. According to the shape of the edge contour points in the input leaf image, the translated mesh model is deformed accordingly to generate a three-dimensional model of the input leaf.

7. A plant leaf three-dimensional modeling system based on a three-dimensional network database, characterized in that: Implementing the plant leaf three-dimensional modeling method based on a three-dimensional network database as described in any one of claims 1 to 6, the system comprises: a database construction module, a leaf prediction module, a local feature extraction module, a global feature extraction module, a model initialization module, and a leaf three-dimensional deformation module: The database construction module collects leaves of various types and at different times, and scans the leaves through a 3D scanner to construct a three-dimensional network database of multiple types of leaves; The blade prediction module is used to obtain the classification information of the input blade, and find the grid model of the required deformation in the three-dimensional network database of multiple types of blades based on the classification information; The local feature extraction module generates main vein feature information according to the input leaf image, and generates a main vein trend function based on the main vein feature information; The global feature extraction module extracts feature information of the edge contour of the blade according to the input blade image, and generates a point set of the edge contour based on the feature information of the edge contour of the blade; The model initialization module translates the mesh model based on the mesh model selected in the leaf prediction module so that the petiole point is located at the origin, and rotates the model so that the leaf tip point is located at axis, and then project the points on the main veins in the grid model onto the maximum imaging plane; The leaf three-dimensional deformation module is based on the grid model initialized in the model initialization module, and deforms the main veins of the grid model according to the main vein trend function of the input leaf, and maps the edge contour points of the grid model with the edge contour point set of the input leaf, thereby obtaining the three-dimensional model of the input leaf.

Images