A data synthesis and enhancement method based on digital twinning and generative AI

Abstract

This invention belongs to the field of computer vision technology and discloses a data synthesis and enhancement method based on digital twins and generative AI. It constructs a digital plant model through 3D image reconstruction, utilizing multi-view images and computer vision technology. A dataset is synthesized based on the 3D model. Two-dimensional images and annotations are obtained through 3D graphics software design and rendering. The synthesized dataset is then tested, and a 3D plant model is generated using generative AI. The synthesized data is used for training. This invention utilizes computer-generated data enhancement to solve the problem of insufficient training data for deep learning algorithms such as image segmentation. Furthermore, it enhances the synthesized data using generative artificial intelligence and digital twin technology. It constructs a digital model of a real-world plant using 3D reconstruction technology and generates a 3D digital plant model using generative AI technology. This method of generating synthesized data solves the problem of difficulty in obtaining training data in image segmentation tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a data synthesis and enhancement method based on digital twins and generative AI. Background Technology

[0002] In recent years, computer vision technology has been widely applied in agricultural research in my country, including pest and disease detection, plant and fruit identification, and crop and weed detection and classification. Leaf segmentation of plant images has important applications in plant phenotypic analysis and leaf area index measurement. For research based on the National Key Research and Development Program of China (Project No.: 2022YFF0503900), the quality and quantity of the training dataset directly affect the performance of deep learning image segmentation algorithms. However, in some agricultural fields, there are problems such as a limited number of available labeled datasets, small dataset size, and limited dataset variety. Furthermore, data collection and labeling often require significant manpower, material resources, and time.

[0003] Digital twin technology can create virtual models of physical objects in a computer. During dataset collection, the data collection process can be simplified by applying data collection operations to the virtual model, facilitating large-scale, high-precision, and highly flexible data collection. Computer-generated data augmentation methods can address the lack of training data for deep learning algorithms such as image segmentation. Furthermore, generative artificial intelligence and digital twin technology provide effective means for synthetic data augmentation.

[0004] Therefore, a data synthesis and enhancement method based on digital twins and generative AI is proposed to solve the problem of difficulty in obtaining training data in image segmentation tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a data synthesis and enhancement method based on digital twins and generative AI to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a data synthesis and enhancement method based on digital twins and generative AI, comprising the following steps:

[0007] S1: A healthy plant was photographed from multiple angles to obtain a preset number of images, which were then used to reconstruct its 3D model.

[0008] S2: Based on computer vision methods, SIFT feature descriptors are used to describe multiple feature points in each 2D image, and the same feature points are matched between multiple images to complete the pose estimation of the image and the camera model. The 3D structure is established through incremental location cloud registration to obtain sparse point cloud.

[0009] S3: Based on the camera pose information and point cloud position and color information estimated by the SfM method, the three-dimensional geometric structure, color and normal information of the scene represented by dense point cloud are obtained by using multi-view geometric methods. After noise filtering and downsampling operations are performed using a three-dimensional data processing library, the plant part in the scene is extracted.

[0010] S4: Using 3D modeling software, manually create a mesh model of the corn based on a dense point cloud model;

[0011] S5: Design the dataset, build a virtual scene, and then segment the dataset according to the corresponding annotations;

[0012] S6: Use an image segmentation model for training tasks and pre-train using a dataset. Use real plant field images as the test set and a portion of the training set, and manually label the test set. Use this as the standard result to judge the accuracy of the model's task completion. Use the IoU value as an indicator to judge the similarity of the segmentation results, and use the SAM segmentation large model as a comparison of the segmentation model's performance.

[0013] S7: Use generative models to synthesize data, build a three-dimensional model of the plant, and conduct tests.

[0014] Preferably, in step S1, each image has a resolution of 12MP, an aspect ratio of 4:3, and a preset number of 170.

[0015] Preferably, in step S3, the three-dimensional data processing library is Open3D.

[0016] Preferably, in step S4, the 3D modeling software is Blender.

[0017] Preferably, in step S6, the image segmentation model is Unet++, and the dataset is ImageNet.

[0018] Preferably, in step S6, the IoU is the intersection-union ratio of the areas of the segmented results. The larger the value of IoU, the closer the two segmented results are.

[0019] Preferably, in step S6, the SAM is an image segmentation model with zero-sample generalization capability.

[0020] Preferably, step S4 specifically includes the following steps:

[0021] S41: Perform blade fitting;

[0022] S42: Perform stem fitting;

[0023] S43: Fit other parts of the plant.

[0024] Preferably, step S41 specifically includes the following steps:

[0025] Feature points are selected from the point cloud, and the leaves of the plant are reconstructed based on the feature points. A preset number of feature points are selected at a preset interval along the direction of the leaf main vein, perpendicular to the direction of the leaf vein. Then, on the curved surface of the leaf, four adjacent feature points are connected to form a quadrilateral plane.

[0026] Preferably, step S42 specifically includes the following steps:

[0027] The plant stem is fitted with segmented cylindrical surfaces. Cylinders of different radii are used at different locations to match the thickness variations of the original plant stem. The cylindrical surfaces at each location are then linearly and smoothly connected.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] This invention utilizes a computer-generated data augmentation method to address the problem of insufficient training data for deep learning algorithms such as image segmentation. Furthermore, it enhances the synthetic data through generative artificial intelligence and digital twin technology, constructs digital models of real-world plants using 3D reconstruction technology, and generates 3D digital models of plants using generative AI technology. This method of generating synthetic data solves the problem of difficulty in obtaining training data in image segmentation tasks. Attached Figure Description

[0030] Figure 1 This is a flowchart of the present invention;

[0031] Figure 2 This is a schematic diagram of image samples collected in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of a dense point cloud according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of a plant after processing by the Open3D 3D data processing library of this invention.

[0034] Figure 5 This is a schematic diagram of the mesh model reconstruction of the present invention;

[0035] Figure 6 This is a schematic diagram illustrating the test results of the model trained on both real and synthetic datasets according to the present invention.

[0036] Figure 7 This is a schematic diagram of the standard results manually annotated in this invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0040] Please see Figures 1-7 As shown, the present invention provides the following technical solution:

[0041] Example 1

[0042] A data synthesis and enhancement method based on digital twins and generative AI includes the following steps:

[0043] In this invention, a digital twin of the plant is first constructed using real-world plant images; that is, a digital model of the plant is built through 3D image reconstruction. This invention proposes using multi-view images of the plant and employing computer vision and graphics methods to perform 3D reconstruction of the original plant's geometric structure. A point cloud model and a mesh model of the plant are then constructed.

[0044] Then, a dataset based on the 3D model of the plant was synthesized. Computer 3D graphics software was used to design and adjust the 3D scene containing the plant model, and the software's image rendering function was used to obtain a dataset consisting of 2D images of the labeled plants and their corresponding annotations.

[0045] In this invention, the application scenarios of synthetic datasets are explored and their application effects are tested. Based on the advantages of synthetic datasets, a deep learning task for computer vision is designed. The synthetic dataset is then used as its training data to test and demonstrate its role in the training process.

[0046] Finally, a 3D model of the plant was generated using a generative AI model. The above-mentioned dataset synthesis method was used for model annotation and dataset synthesis. The synthesized data obtained from the generative AI model was used in the training process of computer vision deep learning tasks, and the promoting effect of the synthesized data on the training effect was tested and shown.

[0047] To aid in understanding this invention, the specific implementation method and results are detailed below:

[0048] 1. Establish a three-dimensional model of the corn plant. The specific process is as follows:

[0049] 1.1 Image Acquisition of Maize Plants

[0050] Multiple images of a healthy, mature corn plant were taken from various angles to reconstruct its 3D model. The corn plant was placed indoors and transplanted into a flowerpot. Each image has a resolution of 12 MP and an aspect ratio of 4:3 (4027 × 3020 pixels), totaling 170 images. A sample of the image set is shown below. Figure 2 As shown:

[0051] 1.2 Reconstructing the 3D Structure of the Plant: In this invention, the open-source 3D reconstruction toolchain Colmap and the 3D data processing library Open3D are used to reconstruct the 3D structure of the plant model. The specific process is as follows:

[0052] 1.2.1 Reconstructing Sparse Point Clouds

[0053] Structure from Motion is a method for estimating the 3D structure of a scene based on a set of multi-view 2D images. It is based on computer vision methods, using descriptors such as SIFT features to describe multiple feature points in each 2D image, and matching the same feature points across multiple images to estimate the pose of the images and the camera model. Then, incremental location cloud registration is used to establish the 3D structure, resulting in a sparse point cloud.

[0054] This invention uses the incremental SfM method provided by the Colmap toolchain to reconstruct the three-dimensional structure of maize plants. The visualization result of the reconstructed point cloud is shown in the figure below. It includes the location and color information of feature points and the camera pose information estimated by the algorithm.

[0055] 1.2.2 Reconstructing Dense Point Clouds

[0056] This invention uses Colmap's Multi-View Stereo tool. Based on the camera pose information and point cloud position and color information estimated by the SfM method, it uses a multi-view geometric method to obtain the 3D geometric structure, color, and normal information of the scene represented by dense point clouds. The resulting dense point cloud visualization is shown below. Figure 3 As shown;

[0057] Then, after using the 3D data processing library Open3D for noise filtering and downsampling, the corn plant portion of the scene was extracted, as shown in the following figure. Figure 4 As shown

[0058] 1.2.3 Reconstructing the Mesh Model

[0059] This invention uses the 3D modeling software Blender to manually create a corn mesh model based on a dense point cloud model. This simplifies the number of meshes in the model, improves dataset rendering efficiency, and reduces storage space requirements. The steps and effects are as follows:

[0060] (1) Leaf Fitting: Feature points are selected from the point cloud, and the plant's leaves are reconstructed based on these feature points. Along the direction of the leaf midrib, an appropriate number of feature points are selected at regular intervals, perpendicular to the leaf vein direction (1-5 points per group in the example). Then, on the leaf surface, four adjacent feature points are connected to form a quadrilateral plane, such as... Figure 5 As shown on the left.

[0061] (2) Stem Fitting: Segmented cylindrical surfaces are used to fit the corn stems. Cylinders of different radii are used at different locations to match the thickness variations of the original plant stems. Then, the cylindrical surfaces at each location are linearly and smoothly connected, such as... Figure 5 As shown in the image.

[0062] (3) Other parts: For organ models that are difficult to fit manually, such as corn cobs, corn stamens, and corn roots, we choose to purchase corn models created by designers online, make minor adjustments to existing models, and then transplant them into the model. Figure 5 As shown on the right.

[0063] In addition, in this invention, after the three-dimensional model of the corn plant is established, the following steps need to be performed:

[0064] 2. The design, acquisition, and testing of the synthetic dataset are detailed below:

[0065] 2.1 Design of Synthetic Dataset and Construction of Virtual Scene: In this test, the constructed dataset is planned to be used for training the leaf segmentation task model. Therefore, a real-world cornfield dataset of a specific scene was obtained, and based on this real-world dataset, an overall 3D simulation scene was constructed, and a suitable synthesis scheme for the leaf segmentation task dataset was designed. The specific process is as follows:

[0066] 2.1.1 Dataset Design: Considering the realities of data collection, four simulated data collection schemes were designed:

[0067] Option 1: There is only one corn plant in the scene. Collect images of a local area of ​​the plant from different directions and heights, for a total of 200 images.

[0068] Option 2: There is only one corn plant in the scene. Collect 200 images of the plant from different directions and heights.

[0069] Option 3: Randomly distribute multiple corn plants in the scene, take pictures around one plant as the center, and collect images of multiple plants centered on the same plant, for a total of 100 images.

[0070] Option 4: Multiple corn plants are randomly distributed in the scene. Cameras are positioned in a 5x5 square to capture images from top to bottom in the cornfield. Each camera rotates uniformly around an axis perpendicular to the ground, capturing orthogonal projection images from multiple angles and points, for a total of 250 images.

[0071] The dataset samples for various schemes are shown in the table below:

[0072]

[0073] 2.1.2 Build a 3D scene, remove the flower pots from the model, and add a land model, ambient light, and sky model.

[0074] Sky: Create a sky model with ambient light using Blender's Dynamic Sky plugin.

[0075] Land: A planar mesh is used as the geometric model of the land, and a suitable 3D land material with weeds is obtained from the material library and applied to the geometric model to obtain the land model.

[0076] 2.1.3 Labels corresponding to the segmented dataset:

[0077] By observing the built model, it is easy to see that by simply modifying the materials corresponding to each part, thereby changing the color of the corresponding position during rendering, and performing image rendering in the same way as the dataset, the segmentation annotation in the form of a mask can be obtained.

[0078] In this test, we set the leaf pixels to white and the non-leaf pixels to black. The sample annotations for each dataset are shown in the table below:

[0079]

[0080] 2.2 Test Synthetic Dataset

[0081] The image segmentation model Unet++ was used for training, and the ImageNet dataset was used for pre-training. Real-world cornfield images were used as the test set and a portion of the training set. The test set was manually labeled, and this labeling was used as the standard result to judge the model's task accuracy. Intersection over Union (IoU) was used as the metric to judge the similarity of the segmentation results. IoU refers to the intersection-over-union ratio of the segmented areas; the higher the IoU value, the closer the two segmentation results are.

[0082] We also used SAM (Segmentation Model A) as a benchmark for our segmentation model's performance. SAM is an image segmentation model with extremely strong zero-shot generalization ability, typically performing well in segmentation tasks. We uploaded test images to the SAM platform and used the interactive segmentation interface to call the SAM model to segment our images. We performed the corn leaf segmentation task by clicking on the corn leaves in the test images and selected the best results possible.

[0083] The test results are shown in the table below:

[0084]

[0085] The test results show that the model trained using both real and synthetic datasets performs best. SAM, as a general-purpose segmentation model, also performs exceptionally well. The model trained solely on the synthetic dataset is inferior to the model trained on the real dataset, but still shows a significant improvement in segmentation performance compared to the model trained only on the synthetic dataset.

[0086] The visualization of the results revealed that the models trained on the two datasets failed to identify many open areas surrounded by leaves compared to the real-world situation. Considering that no dataset specifically designed for this scenario was provided, future work could consider setting up a targeted dataset for training.

[0087] Furthermore, in this invention, after the design, acquisition, and testing of the synthetic dataset are completed, data synthesis is required through a generative model.

[0088] 3. Data synthesis using generative models: In this invention, taking the Meshy 2 generative AI model (a text-generated model released by the Meshy team) as an example, a 3D model of corn was synthesized. The Meshy model can directly output a textured 3D mesh model. Using this generated 3D model instead of the 3D model created using 3D modeling software, another batch of synthesized data of the same specifications can be obtained through the above data synthesis process. The specific process is as follows:

[0089] 3.1 Establishing a three-dimensional model of a maize plant

[0090] The Meshy 2 model's user interface is used. This invention uses text prompts to guide the generative AI model in generating 3D models.

[0091] The synthetic data was used to train a computer vision deep learning task, and the test results are shown in the table below:

[0092]

[0093] Analysis of the test data shows that the training performance of the dataset generated by the Meshy 2 synthetic model is lower than that of the dataset generated by the model manually built using 3D modeling software. This result is expected given the significant difference in model quality between the two. However, when comparing large-scale SAM segmentation models, the training performance of the dataset generated by the Meshy 2 synthetic model is still higher than the segmentation results of the large-scale SAM segmentation model.

[0094] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0095] Memory, used to store computer programs;

[0096] The processor, when executing a program stored in memory, implements the big data-based intelligent rating and evaluation optimization method of the present invention.

[0097] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage system located remotely from the aforementioned processor.

[0098] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] To aid in understanding the present invention, the following embodiments are also provided:

[0100] Example 2

[0101] In the field of precision agriculture, real-time monitoring of crop health is crucial for increasing crop yields; therefore, the following methods are provided:

[0102] Data Acquisition: High-resolution cameras (such as HD cameras mounted on drones) were used to take multi-angle photos of the cornfield, acquiring 170 images with a resolution of 12MP and an aspect ratio of 4:3.

[0103] 3D Reconstruction: The SIFT algorithm was applied to process the image, and a 3D sparse point cloud of maize plants was established using SfM and incremental point cloud registration techniques.

[0104] Denseization and segmentation: Open3D is used to denoise and downsample the point cloud, and then convert it into a dense point cloud to segment out the corn plant parts.

[0105] 3D modeling: In Blender, a mesh model of the corn plant is manually constructed based on dense point cloud data, including detailed structures such as leaves and stems.

[0106] Dataset preparation: Design a dataset of maize plant images containing different growth stages and health conditions, and augment and annotate them in a virtual environment.

[0107] Model training: Unet++ was used as the image segmentation model. After pre-training on ImageNet, it was fine-tuned using actual field corn image data.

[0108] Performance evaluation: The model's performance on the test set is evaluated using IoU values, and its segmentation accuracy is verified by comparing it with the SAM model.

[0109] Data synthesis and testing: Generative AI models (such as GANs) are used to synthesize more diverse maize plant images to further test and optimize the segmentation model.

[0110] Example 3

[0111] In plant pathology research, the ability to quickly and accurately identify crop diseases is key to developing effective control strategies. The following methods are provided:

[0112] Extension based on Example 2:

[0113] Disease sample collection: Increase the number of photos taken of maize plants with different disease characteristics to ensure that the dataset contains a variety of disease types.

[0114] Lesion annotation: Accurately annotate lesion regions in the dataset so that the model can learn lesion features.

[0115] Model tuning: During training, the learning parameters of the Unet++ model are adjusted to increase the focus on lesion areas and improve recognition accuracy.

[0116] Disease simulation: Using generative AI technology, different disease conditions are simulated on a 3D model of a healthy plant to generate synthetic images and enhance the model's generalization ability.

[0117] Field testing: The trained model is deployed in the field to detect crop lesions in real time and compared with traditional methods to evaluate its effectiveness in practical applications.

[0118] Example 4

[0119] In the construction of smart cities, green coverage and plant health are important indicators for assessing the urban ecological environment. Therefore, the following implementation methods are provided:

[0120] Urban greening data collection: Images of plants in urban green spaces are captured using urban surveillance cameras or drones, and similar methods are applied for 3D reconstruction and segmentation.

[0121] Diverse model training: Not limited to corn, but also requires the construction of 3D models and segmentation models of various plants such as trees, shrubs, and flowers.

[0122] Greening Assessment System: In conjunction with a GIS system, the segmentation results are mapped onto a city map to achieve automatic calculation of green coverage rate and real-time monitoring of plant health status.

[0123] Data-driven decision-making: Based on monitoring results, provide data support for urban greening management and guide greening renovation and plant maintenance.

[0124] Example 5

[0125] Using digital twin technology to build virtual farms provides a platform for agricultural education and research, and the following methods are offered:

[0126] Building a virtual environment: Based on the shooting data of real farms, a high-precision 3D virtual farm environment is constructed.

[0127] Dynamic simulation: Using generative AI technology, it simulates various stages in the crop growth cycle, including germination, growth, flowering, and fruiting.

[0128] User interaction: Integrate interactive functions into the virtual farm, such as planting, watering, and fertilizing, to provide an immersive learning experience.

[0129] Educational applications: Virtual farms are used in agricultural teaching to help students understand agricultural knowledge and techniques through hands-on practice.

[0130] Example 6

[0131] In the field of plant breeding, analyzing the genetic diversity of different varieties is crucial for cultivating new varieties that are high-yielding, disease-resistant, and stress-tolerant. Therefore, the following methods are provided:

[0132] Sample collection and 3D reconstruction: 3D reconstruction of multiple different crop varieties was performed to obtain detailed morphological features.

[0133] Feature extraction and comparison: Using computer vision and machine learning algorithms, morphological features (such as leaf shape, stem thickness, etc.) are extracted from the 3D model and cross-variety comparative analysis is performed.

[0134] Genetic diversity assessment: Combining genomic data to assess the genetic distance and diversity levels between different varieties, providing a scientific basis for breeding work.

[0135] New variety development: Based on the results of genetic diversity analysis, select parents with excellent traits for hybridization breeding to develop new varieties with excellent characteristics such as high yield and disease resistance.

[0136] Example 7

[0137] Using virtual reality (VR) technology to provide the public with immersive agricultural experiences and enhance their understanding and awareness of agricultural production and culture, the following methods are offered:

[0138] Virtual scene construction: Based on the shooting data of real farms and the results of 3D reconstruction, a high-precision virtual agricultural scene is constructed.

[0139] Interactive element design: Design interactive elements in the virtual scene, such as operable farm machinery and harvestable fruits, to increase the fun and interactivity of the user experience.

[0140] Educational content integration: Integrating agricultural knowledge, production processes, cultural heritage, and other content into virtual scenarios and teaching them to users in a gamified way.

[0141] Multi-platform release: Release virtual reality agricultural experience projects to multiple platforms (such as VR headsets, mobile apps, web pages, etc.) for different user groups to experience and learn from.

[0142] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data synthesis and enhancement method based on digital twins and generative AI of the present invention.

[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable vehicles (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "" and / or "" indicate that either one or both can be selected. Furthermore, the terms "includes," "contains," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the statement "includes a..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

[0149] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data synthesis and enhancement method based on digital twins and generative AI, characterized in that: Includes the following steps: S1: A healthy plant was photographed from multiple angles to obtain a preset number of images, which were used to reconstruct its three-dimensional model. The plant was a corn plant. S2: Based on computer vision methods, SIFT feature descriptors are used to describe multiple feature points in each 2D image, and the same feature points are matched between multiple images to complete the pose estimation of the image and the camera model. The 3D structure is established through incremental location cloud registration to obtain sparse point cloud. S3: Based on the camera pose information and point cloud position and color information estimated by the SfM method, the three-dimensional geometric structure, color and normal information of the scene represented by dense point cloud are obtained by using multi-view geometric methods. After noise filtering and downsampling operations are performed using a three-dimensional data processing library, the plant part in the scene is extracted. S4: Using 3D modeling software, manually create a mesh model of the corn based on a dense point cloud model; S5: Design the dataset, build a virtual scene, and then segment the dataset according to the corresponding annotations; S6: Use an image segmentation model for training tasks and pre-train using a dataset. Use real plant field images as the test set and a portion of the training set, and manually label the test set. Use this as the standard result to judge the accuracy of the model's task completion. Use the IoU value as an indicator to judge the similarity of the segmentation results, and use the SAM segmentation large model as a comparison of the segmentation model's performance. S7: Use generative models to synthesize data, build a three-dimensional model of the plant, and test it; S4 specifically includes the following steps: S41: Perform blade fitting; S42: Perform stem fitting; S43: Fit other parts of the plant; S41 specifically includes the following steps: Feature points are selected from the point cloud, and the leaves of the plant are reconstructed based on the feature points. A preset number of feature points are selected at a preset interval along the direction of the leaf main vein, perpendicular to the direction of the leaf vein. Then, on the curved surface of the leaf, four adjacent feature points are connected to form a quadrilateral plane. S42 specifically includes the following steps: The plant stem is fitted with segmented cylindrical surfaces. Cylinders of different radii are used at different locations to match the thickness variations of the original plant stem. The cylindrical surfaces at each location are then linearly and smoothly connected.

2. The data synthesis and enhancement method based on digital twins and generative AI according to claim 1, characterized in that: In S1, each image has a resolution of 12MP, an aspect ratio of 4:3, and a preset number of 170.

3. The data synthesis and enhancement method based on digital twins and generative AI according to claim 1, characterized in that: In S3, the three-dimensional data processing library is Open3D.

4. The data synthesis and enhancement method based on digital twins and generative AI according to claim 1, characterized in that: In step S4, the 3D modeling software is Blender.

5. The data synthesis and enhancement method based on digital twins and generative AI according to claim 1, characterized in that: In step S6, the image segmentation model is Unet++, and the dataset is ImageNet.

6. The data synthesis and enhancement method based on digital twins and generative AI according to claim 1, characterized in that: In step S6, IoU is the intersection-union ratio of the areas of the segmented results. The larger the value of IoU, the closer the two segmented results are.

7. The data synthesis and enhancement method based on digital twins and generative AI according to claim 1, characterized in that: In S6, SAM is an image segmentation model with zero-sample generalization capability.

Images