Plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network

By combining augmented reality technology with graph convolution network in plant organ calibration, intelligent calibration and collaborative operation in three-dimensional space is achieved, the efficiency and accuracy of traditional calibration methods are solved, and more intuitive and efficient plant organs are provided Calibration solution.

CN118982586APending Publication Date: 2024-11-19DONGHUA UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411015962.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Traditional plant organ calibration methods rely on manual observations, with low efficiency and limited accuracy and repeatability. Graph convolutional networks lack intuitiveness and user control in direct push learning, making them difficult to effectively apply in field experiments and on-site monitoring.

Method used

Combining augmented reality technology and graph convolution network, through the plant point cloud calibration platform, processing platform and collaborative operation platform, plant point cloud organ calibration and label proliferation in augmented reality three-dimensional space, allowing users to correct and verify calibration results in real time. Provides a third-party perspective to improve calibration efficiency and accuracy.

Benefits of technology

It significantly improves the efficiency and accuracy of plant organ calibration, overcomes environmental limitations, and realizes accurate calibration that is difficult to achieve in 2D planes, supports real-time operations in the fields and on-site, dynamically generates plant point cloud models, and reduces computing resource usage. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982586A_ABST
    Figure CN118982586A_ABST
Patent Text Reader

Abstract

The invention relates to a plant point cloud organ intelligent calibration system based on augmented reality and a graph convolutional network. Comprising a plant point cloud calibration platform, a plant point cloud processing platform and a cooperative operation platform. A plant point cloud calibration platform renders a point cloud file in a database into a point cloud model in real time, an augmented reality device can be used to manually operate on the point cloud, and calibration of partial organs is performed on the plant point cloud model through a point cloud calibration module of the device; the plant point cloud processing platform provides point cloud analysis, label reasoning and data transmission services for the plant point cloud calibration platform; the cooperative operation platform enables an operator to jump out of the limitation of equipment, and enables a third party to participate in a plant point cloud organ segmentation task through mobile terminal equipment at the same time. According to the method, plant point cloud organ label reasoning and organ instance segmentation in an AR space are realized, the operation environment limitation of a traditional organ calibration method is solved, and meanwhile, the efficiency and accuracy of organ calibration are improved through immersive operation experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of agricultural engineering, artificial intelligence and augmented reality, and in particular to an intelligent calibration system for plant point cloud organs based on augmented reality and graph convolutional networks. Background Art

[0002] With the growth of global population and the challenge of climate change, the importance of plant phenotyping has become increasingly prominent. In the cultivation of new crop varieties, whether using hybrid breeding or transgenic breeding, it is ultimately implemented in the comparison of new varieties and benchmark varieties in the test field. However, it is a time-consuming and laborious process to manually measure the phenotypic characteristics of all plants in the new variety block in the test field, such as plant height, leaf area, leaf length, leaf width, fruit size, and fruit number. It also requires practitioners to have high knowledge of agricultural products, which seriously restricts the efficiency of breeding. The use of three-dimensional imaging, computer vision and artificial intelligence technology to achieve three-dimensional automatic phenotyping of crops has become the latest development direction in the field of breeding. In fact, the premise of phenotypic analysis is to be able to effectively segment all organs of the plant in three-dimensional space, and the essence of segmentation is actually to effectively calibrate the point cloud covered by each organ. The three-dimensional point cloud plant organ calibration technology can quickly and accurately identify and classify plant organs such as leaves, stems, flowers and fruits through high-precision point cloud processing and analysis.

[0003] In plant breeding, by calibrating and comparing organs of different varieties or genotypes of plants, researchers can screen out excellent genetic traits. In agricultural production, organ calibration technology can be used for crop disease monitoring and yield prediction, helping farmers manage farmland more effectively. In addition, this technology can also be used to monitor the response of plants to environmental changes, providing data support for ecosystem management and the formulation of climate change adaptation strategies.

[0004] In traditional plant phenotyping studies, the calibration of plant organs mostly relies on manual observation and measurement, which is not only inefficient, but also limits the accuracy of the results and the repeatability of the experiment. With the development of deep learning in recent years, sparse calibration and semi-supervised transductive graph convolutional networks can quickly and accurately calibrate many organs in three-dimensional plant point clouds. Compared with traditional manual observation and measurement, graph convolutional networks can effectively learn the local and global features of each point in the point cloud by performing convolution operations on graph structure data, thereby improving the accuracy of identifying and classifying plant organs. In addition, graph convolutional networks can extract useful feature representations from complex point cloud data, which is crucial for understanding and analyzing the three-dimensional structure of plants. Compared with traditional manual calibration methods, the graph convolutional network calibration technology combined with sparse labeled point clouds can significantly improve efficiency and accuracy. However, there are also some limitations to this. The process of transductive learning of sparsely labeled point clouds by graph convolutional networks is not particularly intuitive. Traditional graph convolutional methods are usually trained and tested entirely by the network at one time in plant organ calibration, which means that users have limited control and intervention in the calibration process. Therefore, this led us to think that combining augmented reality technology with the label reasoning process of graph convolution can make up for the shortcomings of insufficient visualization and human-computer interaction, and bring more intuitive automatic tools to plant 3D phenotyping.

[0005] With the development of augmented reality technology, virtual information can be combined with the real physical world. Augmented reality can overcome the limitations of stereoscopic reality in traditional semi-supervised point cloud calibration, and allow users to participate in the calibration process in a more intuitive way with rich three-dimensional user interaction tools, and real-time correction and verification of point cloud calibration and segmentation results. And augmented reality technology allows operators to jump out of the limitations of laboratory and computer environments, so that they can calibrate plant organs in any occasion, which is particularly important for field trials and on-site monitoring.

[0006] The plant organ calibration method that combines augmented reality with graph convolutional networks not only retains the advantages of GCN in processing complex data structures, but also increases interactivity, intuitiveness, and real-time performance, making the calibration process more efficient and accurate. The development of this method brings new possibilities for plant phenotyping and precision agriculture. Summary of the invention

[0007] The purpose of the present invention is to provide a plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network. Through the communication and collaboration between the plant point cloud calibration platform, the plant point cloud processing platform and the collaborative operation platform, the plant point cloud organ calibration, label proliferation and organ segmentation in the augmented reality three-dimensional space are realized, the environmental limitations of the traditional organ calibration method are solved, and the efficiency of organ calibration and segmentation accuracy are improved.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network includes three parts: a plant point cloud calibration platform, a plant point cloud processing platform, and a collaborative operation platform. The plant point cloud calibration platform is responsible for receiving the unlabeled plant point cloud file sent by the plant point cloud processing platform, generating an unlabeled plant point cloud model by using a plant point cloud rendering module, and performing organ calibration by using a plant point cloud calibration module. For the point cloud result after completing label reasoning, the organ of the plant point cloud can be segmented and restored by using a plant organ separation module; the plant point cloud processing platform is responsible for sending the unlabeled plant point cloud file to the plant point cloud calibration platform and receiving the plant point cloud file with local labels sent back by the plant point cloud calibration platform, and then automatically performing label reasoning of all points by using a graph convolution label reasoning module, and finally returning the result to the plant point cloud calibration platform for display; the collaborative operation platform provides a real-time third-party perspective for the calibration operation of the plant point cloud, and allows another operator holding the collaborative operation platform to participate in the plant point cloud organ calibration task, and perform real-time synchronous operation with the plant point cloud calibration platform for the same plant point cloud model.

[0010] The present invention has the following beneficial effects and advantages:

[0011] 1. The use of the present invention requires a simple environment. As long as there is an AR device and a PC, the semi-supervised calibration of plant organ examples can be performed, which solves the environmental limitations of traditional organ calibration methods and can significantly improve the calibration efficiency of plant organs;

[0012] 2. The present invention can dynamically generate a plant point cloud model without having to save the point cloud model as a graphical prefabricated part in advance. Even after zooming in, each point will be clearly visible. Since the three-dimensional model generated in the field of view of the AR device is implemented using an instant rendering strategy, it will not occupy too much computing resources of the device, and point cloud operations can be realized in real time;

[0013] 3. The present invention can perform plant point cloud calibration in real time in augmented reality three-dimensional space, avoiding the difficulty of accurate calibration caused by organ overlap in the 2D plane;

[0014] 4. The present invention can provide a third-person perspective of AR through a mobile device, and operators holding third-party mobile devices can also collaborate in the calibration task, which can significantly improve the calibration efficiency of plant organs;

[0015] 5. The present invention performs semi-supervised reasoning and calibration of plant organs based on graph convolutional networks. The graph convolutional network can effectively learn the local and global features of each point in the point cloud by performing convolution operations on graph structured data. It can work under the condition of only sparse labels and can calibrate and segment all points of organs in the plant point cloud model with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a system structure diagram of the present invention;

[0017] Figure 2 It is a schematic diagram of the use of the present invention.

[0018] The numbers in the figure represent:

[0019] 1. Plant point cloud calibration platform, 2. Plant point cloud processing platform, 3. Collaborative operation platform, 11. Plant point cloud rendering module, 12. Plant point cloud calibration module, 13. Plant organ separation module, 14. AR device, 15. TCP client module, 16. Collaborative operation server module, 17. Function menu, 18. Model fixed button module, 21. Graph convolution label inference module, 22. TCP server module, 23. Hardware device, 31. Mobile device, 32. Collaborative operation client module, 121. Calibration container, 122. Slide bar, 123. Label modification button, 124. Calibration button, 125. Calibration container selection menu, 171. Enter / exit point cloud calibration button, 172. Separate / restore plant point cloud button, 173. Send button, 174. Receive button. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the application.

[0021] like Figure 1As shown, a plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network includes: a plant point cloud calibration platform 1, a plant point cloud processing platform 2, and a collaborative operation platform 3. The point cloud calibration platform 1 is responsible for receiving the unlabeled plant point cloud file sent by the plant point cloud processing platform 2, using the plant point cloud rendering module 11 to generate an unlabeled plant point cloud model, and performing organ partial calibration through the plant point cloud calibration module 12. For the point cloud results that complete the label reasoning, the plant point cloud organs can be segmented and restored through the plant organ separation module 13; the plant point cloud processing platform 2 is responsible for sending the unlabeled plant point cloud file to the plant point cloud calibration platform 1 and receiving the plant point cloud file with local labels sent back by the plant point cloud calibration platform 1, and then automatically performing label inference for all points through the graph convolution label inference module 21, and finally returning the result to the plant point cloud calibration platform 1; the collaborative operation platform 3 provides a real-time third-party perspective for the calibration operation of the plant point cloud, and allows another operator holding the collaborative operation platform 3 to participate in the organ calibration task of the plant point cloud, and perform real-time synchronous operations with the plant point cloud calibration platform 1 for the same plant point cloud model.

[0022] Specifically, the point cloud files used in practice by this system are all PCD format files, but are not limited to the PCD format, and point cloud files such as TXT and PLY can also be used.

[0023] Specifically, the AR device 14 used in the first implementation of the system is a Microsoft HoloLens 2 AR helmet, but the idea of ​​this patent can cover alternative solutions using other brands and similar augmented reality or mixed reality hardware devices.

[0024] Specifically, the plant point cloud calibration platform 1, the plant point cloud processing platform 2, and the collaborative operation platform 3 communicate via a wired connection or wireless Internet.

[0025] Specifically, the hardware device 23 in the plant point cloud processing platform is a PC equipped with an Nvidia RTX3090 GPU when it is first implemented, which can simultaneously perform point cloud data transmission and reception and point cloud semi-supervised GCN label inference calculation under Windows and Ubuntu operating systems. However, the idea of ​​the patent can cover alternative solutions using other brands and similar hardware devices.

[0026] Specifically, the mobile hardware device 31 in the collaborative operation platform is a mobile tablet equipped with the Android system when it is first implemented, but the idea of ​​this patent can cover alternative solutions using other operating systems and similar mobile hardware devices.

[0027] Furthermore, the plant point cloud calibration platform 1 and the plant point cloud processing platform 2 communicate using the TCP communication protocol, and the IP addresses and port numbers need to be configured on the two platforms before use.

[0028] like Figure 1 , Figure 2As shown, the plant point cloud calibration platform 1 of the present invention includes: an AR device 14, a plant point cloud rendering module 11, a plant point cloud calibration module 12, a plant organ separation module 13, a TCP client module 15, a collaborative operation server module 16, a function menu 17, and a model fixing button 18. The AR device 14 is worn by an operator to perform interactive operations on the calibration task of the plant point cloud; the plant point cloud rendering module 11 reads the plant point cloud file sent by the plant point cloud processing platform 2, and saves the point cloud data information to the AR device 14, and at the same time renders it into a plant point cloud model that can be viewed and modified in real time in the center of the field of view of the AR device 14 according to the point cloud information; the plant point cloud calibration module 12 generates a plant point cloud model in the field of view. The bottom is firmly bound to the point cloud and will rotate, scale and move with the plant point cloud model. When the operator is doing organ calibration, he will label the organs of the plant point cloud model through a calibration container 121 in the calibration module, and the color of the labeled points will change in real time; the plant organ separation module 13 can not only separate the organs of the plant point cloud model that has completed organ calibration into separate organ models, but also merge the separated separate organ models back into the overall plant point cloud model. The separated organ models can be rotated, scaled and moved respectively, so as to facilitate the observation of the characteristics of each separated organ; the TCP client module 15 interacts with the point cloud processing platform to complete TCP network communication, including receiving unlabeled plant object point cloud file, sending plant point cloud files with partial labels, and receiving plant point cloud files with completed label inference; the collaborative operation server module 16 synchronizes the operation of the operator wearing the AR device 14 on the plant point cloud to the operator holding the collaborative operation platform 3, thereby providing a third perspective of the plant point cloud model observed in the AR device 14, and allowing the operator holding the collaborative operation platform 3 to participate in the plant point cloud calibration task; the function menu 17 is located above the operator's field of view and will move with the operator. The 4 buttons on the menu are, from left to right, enter / exit point cloud calibration button 171, separate / restore plant point cloud button 172, send button 173, receive button 174, enter / exit point cloud After the calibration button 171 is clicked, if there is no plant point cloud calibration module 12 in the field of view at this time, the plant point cloud calibration module will be generated below the plant point cloud model, otherwise the plant point cloud calibration module 12 will be destroyed, but the labeled plant point cloud will not be changed; after the separation / restore plant point cloud button 172 is clicked, the plant organ separation module 13 will be called to separate the plant organs. If the plant organs have been separated, the separated plant organs will be reassembled and restored to the overall plant point cloud model; after the send button 173 is clicked, the plant point cloud file with partial labels will be sent to the plant point cloud processing platform 2; after the receive button 174 is clicked, the result of the graph convolutional network label inference will be received from the plant point cloud processing platform 2;The model fix button 18 is fixed at the upper right corner of the point cloud model, and rotates, scales, and moves with the point cloud model. If the button is pressed, it will prohibit / allow the movement, rotation, and scaling of the point cloud model. ;

[0029] Specifically, by clicking the small icon in the upper right corner of the function menu 17 , the function menu 17 can be moved with the operator, and by clicking again, the function menu 17 can be fixed at the current position.

[0030] Specifically, when the plant point cloud calibration platform 1 is started, it will first use the collaborative operation server module 16 to create a room in the cloud, and the TCP connection prompt box will appear only after the creation is successful.

[0031] The plant point cloud rendering module 11 includes the following steps when implemented:

[0032] Step S111: Generate (randomly or according to a pre-stored color table) a list C containing color information = [C1, C2, ..., C T ] T , specifies the color corresponding to each label during plant point cloud calibration, and the colors will not be repeated;

[0033] Step S112: parse the plant point cloud file sent by the plant point cloud processing platform 2, and generate a list to store the coordinate information and label of each point If the plant point cloud file is an unlabeled point cloud file, the labels of these points are regarded as 0;

[0034] Step S113: Use a small cube grid model to render each point in the point cloud, and dynamically generate grid data for organs of each type of label according to the label of the point. Use the color table generated in step S111 to render the model parts corresponding to each label separately. The model of the entire plant point cloud is now visualized in the center of the field of view of the AR device. The plant point cloud model is enclosed by a cuboid. Through the operation of the operator's hands, the cuboid can be rotated, moved, and scaled to achieve real-time rotation, movement, and scaling of the entire plant point cloud model within the field of view.

[0035] Specifically, different models can be used to implement the shape of each point, such as a sphere, a cube, and a cone. However, considering the performance of most AR devices, if the point cloud has a large number of points, choosing a model with fewer triangles will provide a better user experience.

[0036] like Figure 2As shown, the menu formed by the plant point cloud calibration module 12 of the present invention includes five parts: a calibration container 121, a slide bar 122, a label modification button 123, a calibration button 124, and a calibration container selection menu 125. The calibration container 121 is initialized at the center of the plant point cloud model. The color of the container corresponds to the color of the current label, which can be changed in real time with the change of the label color. It can be moved, scaled, and rotated. When the calibration button 124 is pressed, the calibration container 121 will label all the points wrapped by the container in the space, and assign the corresponding colors of the labels to these points in real time (which can correspond to a predefined color table, such as label 1 corresponds to blue and 2 corresponds to red). ; The slide bar 122 is located below the plant point cloud model and will perform the same movement as the model is rotated, scaled, and moved; the label modification button 123 is located at the lower left of the slide bar 122. Pressing this button will modify the label to be added, and the color of the button and the color of the calibration container 121 will also change to the color corresponding to the label; the calibration button 124 is located at the lower right of the slide bar 123. When this button is pressed, the color of the points in the calibration container 121 will change in real time to the color corresponding to the current label; the calibration container selection menu 125 is located on the left side of the plant point cloud model. The operator can replace the calibration container with a rectangular, cylindrical, or spherical geometric body through the menu.

[0037] The present invention modifies the plant point cloud color in real time, comprising the following steps:

[0038] Step S11-1: first determine which model the current calibration container 121 is, and then select different modification algorithms;

[0039] Step S11-2: read the spatial coordinates of the current container in the original plant point cloud coordinate system, traverse each point, calculate whether each point is inside the container, and modify the plant point cloud label list in real time;

[0040] Step S11-3: After pressing the calibration button 124, the current plant point cloud model is destroyed, and a new model is generated according to the plant point cloud label list that changes in real time;

[0041] Step S11-4: modifying the plant point cloud at the current moment according to the position, rotation, and scaling parameters of the plant point cloud at the previous moment;

[0042] The TCP client module 15 comprises the following steps:

[0043] Step S151: When the program starts, after selecting to connect, an asynchronous TCP request is initiated to the plant point cloud processing platform 2. When the TCP connection is completed, the unlabeled point cloud file is received;

[0044] Step S152: When the operator presses the send button 173 of the function menu 17, the point cloud with partial calibration is sent to the plant point cloud processing platform 2 in the form of bytes, and a start reasoning signal is sent to allow the plant point cloud processing platform 2 to start the label reasoning of the point cloud;

[0045] Step S153: When the operator presses the receiving button 174 of the function menu 17, the plant point cloud calibration platform 1 will send an inference result receiving signal to the plant point cloud processing platform 2, and then the plant point cloud calibration platform 1 starts to receive the plant organ label inference result completed by the plant point cloud processing platform 2 through the graph convolutional network;

[0046] Step S154: After the data is received, the previously partially labeled point cloud model is destroyed, and at the same time, the plant point cloud rendering module 11 is used to read the received new point cloud file with all the inference labels, and the new point cloud model is rendered in front of the field of view.

[0047] The collaborative operation server module 16 includes: after the plant point cloud calibration platform 1 starts to establish a connection with the plant point cloud processing platform 2, a room will be created in the cloud, and the operator holding the collaborative operation platform 3 can join the room for collaborative operation; the plant point cloud file received by the plant point cloud calibration platform 1 is synchronized to the collaborative operation platform 3, and the point cloud rendering is synchronized with the collaborative operation platform 3; the plant point cloud calibration platform 1 and the collaborative operation platform 3 synchronize the plant point cloud model, movement, scaling, and calibration operations.

[0048] Specifically, when an operator wearing a HoloLens 2 helmet and an operator using an Android device operate the same object at the same time, the operator wearing the HoloLens 2 helmet will have priority control.

[0049] like Figure 1 As shown, the plant point cloud processing platform 2 includes: a graph convolution label inference module 21, a TCP server module 22, and a PC 23, wherein the graph convolution label inference module 21 performs label inference on the plant point cloud containing only partial labels sent by the plant point cloud calibration platform 1, and uses the graph convolution network to learn and understand the point cloud structure to achieve semi-supervised organ segmentation; the TCP server module 22 processes the communication and file sending and receiving between the PC and the point cloud calibration platform 1. When a connection request from the plant point cloud calibration platform 1 is received, a new thread will be established to respond to the request, and corresponding work will be automatically performed according to different signals sent from the plant point cloud calibration platform 1; the hardware device 23 mounts the TCP server module 22 and the graph convolution label inference module 21, and can automatically process the plant point cloud according to the operation of the plant point cloud calibration platform 1 without manual operation.

[0050] Specifically, before use, the plant point cloud file to be processed can be selected on the plant point cloud processing platform 2 .

[0051] The graph convolution label processing module includes: a data processing unit, a graph construction unit, a network training unit, and a label inference unit. The data processing unit processes the point cloud with partial labels sent by the plant point cloud calibration platform 1. According to the labels, half of the labeled points are set as a training set, generally as a verification set, and the unlabeled points are set as a test set; the graph construction unit constructs an adjacency matrix based on the k-nearest neighbor principle, and returns an adjacency matrix consisting of the true distances between sample points and an adjacency matrix using 0 and 1 to represent the connection status. Then, the point cloud coordinates are spliced ​​with the adjacency matrix representing the connection status to form a new feature matrix; the network training unit uses the new feature matrix to perform network training; the label inference unit uses the trained network to infer the point cloud with partial labels into a point cloud with complete label coverage.

[0052] The graph convolution label processing module includes the following steps:

[0053] Step S1: Use the data processing unit to read the received point cloud file, and divide all points into two categories according to whether each point is labeled, using two lists and To store the indexes of these points;

[0054] Step S2: Using the graph construction unit, construct an adjacency matrix A containing its own connection status based on the k-nearest neighbor principle c and A d , where A c A is composed of 0 and 1, representing the adjacency matrix of the connection state. d It is composed of the real distance between sample points, which indicates both the connection status between points and the distance information between points;

[0055] Step S3: The feature matrix obtained by the data processing unit With A c Concatenate as the new feature matrix F = [A c ,F'], it can be understood that the new feature matrix formed in this way has both the location information of the point itself and the neighborhood correlation, and contains rich input information;

[0056] Step S4: A d In the matrix, sort the non-zero elements of each row from largest to smallest, and set the first half of the elements to 0, thereby removing the longer connecting edges;

[0057] Step S5: Initialize the network in the network training unit, and then use the new feature matrix F and adjacency matrix A d Train the network and list The corresponding points are randomly divided into two halves, one half as a training set and the other half as a validation set;

[0058] Step S6: In the label inference unit, list The corresponding points are used as the test set, and the trained graph convolutional network is used to infer the point cloud with partial labels into a point cloud with complete label coverage, and the inference results are saved.

[0059] like Figure 1 As shown, the collaborative operation platform 3 includes: a mobile terminal device 31 and a collaborative operation client module 32. When the operator wearing the AR device 14 is working, another operator can use the mobile terminal device 31 mounted with the collaborative operation client module 32 to see the operator of the AR device 14's operations on the point cloud in his field of view, and the operator using the collaborative operation platform 3 can also use a similar calibration container 121 to calibrate, move, scale, and rotate the point cloud model by clicking on the screen; when the collaborative operation platform 3 is started, the collaborative operation client module 32 will send a connection request to the plant point cloud calibration module 1 and join the cloud room created by the plant point cloud calibration module 1. After successfully joining, the environment of the plant point cloud calibration module 1 will be synchronized. When the plant point cloud calibration module 1 receives the plant point cloud file, it will be sent to the collaborative operation platform 3 synchronously. The collaborative operation platform 3 can perform point cloud model rendering synchronized with the plant point cloud calibration platform 1 according to the point cloud file.

[0060] Specifically, the collaborative operation platform 3 can only be used after the plant point cloud calibration platform 1 successfully creates a room in the cloud.

[0061] Specifically, the plant point cloud model generated by the mobile device 31 is also located in front of the device. Even if the operator wearing the AR device 14 and the operator holding the mobile device 31 are not in the same room, the point cloud of the same plant can be calibrated.

[0062] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network, characterized in that: include: Plant point cloud calibration platform (1), plant point cloud processing platform (2), collaborative operation platform (3); The plant point cloud calibration platform (1) is responsible for receiving the unlabeled plant point cloud file sent by the plant point cloud processing platform (2), generating an unlabeled plant point cloud model using the plant point cloud rendering module (11), and performing organ calibration through the plant point cloud calibration module (12). For the point cloud result after label inference, the plant organ separation module (13) is used to segment and restore the organs of the plant point cloud; The plant point cloud processing platform (2) is responsible for sending the unlabeled plant point cloud file to the plant point cloud calibration platform (1) and receiving the plant point cloud file with local labels sent back by the plant point cloud calibration platform (1), and then automatically performing label inference for all points through the graph convolution label inference module (21), and finally returning the result to the plant point cloud calibration platform (1) for display; The collaborative operation platform (3) provides a real-time third-party perspective for the calibration operation of the plant point cloud, and allows another operator holding the collaborative operation platform (3) to participate in the organ calibration task of the plant point cloud, and perform real-time synchronous operations with the plant point cloud calibration platform (1) on the same plant point cloud model.

2. According to claim 1, the plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network is characterized in that: The structure of the plant point cloud calibration platform (1) comprises: an AR device (14), a plant point cloud rendering module (11), a plant point cloud calibration module (12), a plant organ separation module (13), a TCP client module (15), a collaborative operation server module (16), a function menu (17), and a model fixing button (18); The AR device (14) is worn by an operator to perform interactive operations for the plant point cloud calibration task; The plant point cloud rendering module (11) reads the plant point cloud file sent by the plant point cloud processing platform (2), saves the point cloud data information to the AR device (14), and renders the point cloud model in the center of the field of view of the AR device (14) according to the point cloud information, which can be viewed and modified in real time; The plant point cloud calibration module (12) will be generated below the plant point cloud model in the field of view, firmly bound to the point cloud, and will rotate, scale, and move with the plant point cloud model. When the operator is calibrating an organ, he or she can label the organ of the plant point cloud model through a calibration container (121) in the calibration module, and the color of the labeled point will change in real time. The plant organ separation module (13) separates the organs of the plant point cloud model that has completed organ calibration into separate organ models, and can also merge the separated separate organ models back into the overall plant point cloud model. The separated organ models can be rotated, scaled, and moved respectively, so as to facilitate the observation of the characteristics of each separated organ. The TCP client module (15) interacts with the point cloud processing platform to complete TCP network communication, including receiving unlabeled plant point cloud files, sending plant point cloud files with partial labels, and receiving plant point cloud files with completed label inference; The collaborative operation server module (16) synchronizes the operation of the plant point cloud by the operator wearing the AR device (14) to the operator holding the collaborative operation platform (3), thereby providing a third perspective of the plant point cloud model observed in the AR device (14) and allowing the operator holding the collaborative operation platform (3) to participate in the calibration task of the plant point cloud; The function menu (17) is located above the operator's field of view and will move with the operator. The four buttons on the menu are, from left to right, the enter / exit point cloud calibration button (171), the separate / restore plant point cloud button (172), the send button (173), and the receive button (174). After the enter / exit point cloud calibration button (171) is clicked, if the plant point cloud calibration module (12) does not exist in the field of view at this time, the plant point cloud calibration module will be generated below the plant point cloud model, otherwise the plant point cloud calibration module (12) will be destroyed. ), but will not change the labeled plant point cloud; after the separate / restore plant point cloud button (172) is clicked, the plant organ separation module (13) will be called to separate the plant organs. If the plant organs have been separated, the separated plant organs will be reassembled and restored to the overall plant point cloud model; after the send button (173) is clicked, the plant point cloud file with some labels will be sent to the plant point cloud processing platform (2); after the receive button (174) is clicked, the result of the graph convolutional network label inference will be received from the plant point cloud processing platform (2); The model fixing button (18) is fixed at the upper right corner of the point cloud model and rotates, scales and moves with the point cloud model. If the button is pressed, it will prohibit / allow the movement, rotation and scaling operations of the point cloud model.

3. According to claim 2, the plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network is characterized in that: The plant point cloud rendering module (11) comprises the following steps when implemented: Step S111: Generate a list C containing color information = [C1, C2, ..., C T ] T , specifies the color corresponding to each label during plant point cloud calibration, and the colors will not be repeated; Step S112: parse the plant point cloud file sent by the plant point cloud processing platform (2) and generate a list to store the coordinate information and label of each point If the plant point cloud file is an unlabeled point cloud file, the labels of these points are regarded as 0; Step S113: Use a small cube grid model to render each point in the point cloud, and dynamically generate grid data of organs of each type of label according to the label of the point; use the color table generated in the step S111 to render the model part corresponding to each label separately; the model of the entire plant point cloud is visualized in the center of the field of view of the AR device at this time, and the plant point cloud model is enveloped by a cuboid. Through the operation of the operator's hands, the cuboid is rotated, moved, and scaled to achieve real-time rotation, movement, and scaling of the entire plant point cloud model within the field of view.

4. According to claim 2, the plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network is characterized in that: The menu formed by the plant point cloud calibration module (12) includes five parts: a calibration container (121), a slide bar (122), a label modification button (123), a calibration button (124), and a calibration container selection menu (125). The calibration container (121) is initialized at the center of the plant point cloud model. The color of the container corresponds to the color of the current label. It changes in real time with the change of the label color and can be moved, scaled, and rotated. When the calibration button (124) is pressed, the calibration container (121) will label all points wrapped by the container in the space and assign the color corresponding to the label to these points in real time. The slide bar (122) is located in the plant point cloud model. The label modification button (123) is located at the lower left of the slider (122). Pressing the button will modify the label to be added, and the color of the button and the color of the calibration container (121) will also change to the color corresponding to the label; the calibration button (124) is located at the lower right of the slider (123). When the button is pressed, the color of the points in the calibration container (121) will change in real time to the color corresponding to the current label; the calibration container selection menu (125) is located on the left side of the plant point cloud model. The operator can replace the calibration container with a rectangular, cylindrical, or spherical geometric body through the menu.

5. According to claim 2, the plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network is characterized in that: The TCP client module (15) comprises the following steps: Step S151: When the program starts, after selecting to connect, an asynchronous TCP request is initiated to the plant point cloud processing platform (2). When the TCP connection is completed, the unlabeled point cloud file is received; Step S152: When the operator presses the send button (173) of the function menu (17), the point cloud with partial calibration is sent to the plant point cloud processing platform (2) in the form of bytes, and a start reasoning signal is sent to allow the plant point cloud processing platform (2) to start the point cloud label reasoning; Step S153: When the operator presses the receiving button (174) of the function menu (17), the plant point cloud calibration platform (1) sends an inference result receiving signal to the plant point cloud processing platform (2), and then the plant point cloud calibration platform (1) starts to receive the plant organ label inference result completed by the plant point cloud processing platform (2) through the graph convolutional network; Step S154: After the data is received, the previously partially labeled point cloud model is destroyed, and at the same time, the plant point cloud rendering module (11) is used to read the received new point cloud file with all the inference labels, and the new point cloud model is rendered in front of the field of view.

6. The plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network according to claim 2 is characterized in that: The collaborative operation server module (16) includes: after the plant point cloud calibration platform (1) starts to establish a connection with the plant point cloud processing platform (2), a room is created in the cloud, and an operator holding a collaborative operation platform (3) can join the room for collaborative operation; the plant point cloud file received by the plant point cloud calibration platform (1) is synchronized to the collaborative operation platform (3), and the point cloud rendering is performed synchronously with the collaborative operation platform (3); the plant point cloud calibration platform (1) and the collaborative operation platform (3) synchronize the plant point cloud model, movement, scaling, and calibration operations.

7. The plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network according to claim 1 is characterized in that: The plant point cloud processing platform (2) comprises: a graph convolution label inference module (21), a TCP server module (22), and a hardware device (23), wherein the graph convolution label inference module (21) performs label inference on the plant point cloud containing only partial labels sent by the plant point cloud calibration platform (1), and uses the graph convolution network to learn and understand the point cloud structure to achieve semi-supervised organ segmentation; the TCP server module (22) processes the communication between the PC and the point cloud calibration platform (1) and the file sending and receiving, and when receiving a connection request from the plant point cloud calibration platform (1), a new thread is established to respond to the request, and corresponding work is automatically performed according to different signals sent by the plant point cloud calibration platform (1); the hardware device (23) mounts the TCP server module (22) and the graph convolution label inference module (21), and can automatically process the plant point cloud according to the operation of the plant point cloud calibration platform (1) without manual operation.

8. The plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network according to claim 7, characterized in that: The graph convolution label processing module comprises: a data processing unit, a graph construction unit, a network training unit, and a label inference unit; the data processing unit processes the point cloud with partial labels sent by the plant point cloud calibration platform (1), and according to the labels, sets half of the labeled points as a training set, the other half as a verification set, and the unlabeled points as a test set; the graph construction unit constructs an adjacency matrix based on the k-nearest neighbor principle, and returns an adjacency matrix consisting of the real distances between sample points and an adjacency matrix representing the connection status with 0 and 1, and then concatenates the point cloud coordinates with the adjacency matrix representing the connection status to form a new feature matrix; the network training unit uses the new feature matrix to perform network training; the label inference unit uses the trained network to infer the point cloud with partial labels into a point cloud with complete label coverage.

9. The plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network according to claim 7, characterized in that: The graph convolution label processing module includes the following steps: Step S1: Use the data processing unit to read the received point cloud file, and divide all points into two categories according to whether each point is labeled, using two lists and To store the indexes of these points; Step S2: Using the graph construction unit, construct an adjacency matrix A containing its own connection status based on the k-nearest neighbor principle c and A d , where A c A is composed of 0 and 1, representing the adjacency matrix of the connection state. d It is composed of the real distance between sample points, which indicates both the connection status between points and the distance information between points; Step S3: The feature matrix obtained by the data processing unit With A c Concatenate as the new feature matrix F = [A c ,F']; Step S4: A d In the matrix, sort the non-zero elements of each row from largest to smallest, and set the first half of the elements to 0, thereby removing the longer connecting edges; Step S5: Initialize the network in the network training unit, and then use the new feature matrix F and adjacency matrix A d Train the network and list The corresponding points are randomly divided into two halves, one half as a training set and the other half as a validation set; Step S6: In the label inference unit, list The corresponding points are used as the test set, and the trained graph convolutional network is used to infer the point cloud with partial labels into a point cloud with complete label coverage, and the inference results are saved.

10. The plant point cloud organ intelligent calibration system based on augmented reality and graph convolutional network according to claim 1, characterized in that: The collaborative operation platform (3) comprises: a mobile terminal device (31) and a collaborative operation client module (32). When an operator wearing an AR device (14) is working, another operator uses a mobile terminal device (31) mounted with the collaborative operation client module (32) to see the operation of the operator of the AR device (14) on the point cloud in his field of view, and the operator using the collaborative operation platform (3) uses the calibration container (121) to calibrate, move, scale and rotate the point cloud model by clicking on the screen; when the collaborative operation platform (3) is started, the collaborative operation client module (32) sends a connection request to the plant point cloud calibration module (1) and joins the cloud room created by the plant point cloud calibration module (1). After joining successfully, the environment of the plant point cloud calibration module (1) is synchronized. When the plant point cloud calibration module (1) receives a plant point cloud file, it is synchronously sent to the collaborative operation platform (3). The collaborative operation platform (3) performs synchronous point cloud model rendering with the plant point cloud calibration platform (1) according to the point cloud file.