Digital twin model correction method and system based on machine vision

Abstract

The invention discloses a digital twin model correction method and system based on machine vision, and relates to the technical field of digital twin model correction. According to the digital twin model correction method based on machine vision, multispectral image data of a plurality of shooting angles of a plurality of plants in a to-be-monitored test planting area are obtained in real time, fusion processing is carried out to obtain a three-dimensional voxel data set of each plant, feature extraction is carried out to obtain a plant feature set of each plant, and the plant feature set of each plant is calculated. Based on the three-dimensional voxel data set and the plant feature set of each plant, performing matching correction processing on the initial digital twinborn model to obtain a comprehensive correction digital twinborn model; according to the method, the comprehensive correction digital twinborn model is analyzed to obtain the real-time health comprehensive index of the regional plants in the test planting region to be monitored, so that the growth state and spatial distribution of the plants are accurately reflected, the model can simulate the real environment more truly, and a more reliable basis is provided for planting management.

Description

Technical Field

[0001] The present invention relates to the technical field of digital twin model correction, and specifically to a digital twin model correction method and system based on machine vision. Background Art

[0002] Machine vision refers to the use of computers and corresponding hardware, such as cameras, sensors, processing units, etc., to simulate the human visual system, thereby acquiring, processing and analyzing image or video data to extract meaningful information. The digital twin model refers to the combination of real-time data of physical entities and virtual simulation models to create a digital copy that can accurately reflect the real object or system. This copy can not only reflect the state of the physical entity in real time, but also with the development of digital technology, Internet of Things, big data, cloud computing and artificial intelligence, digital twin technology is increasingly used in various fields, especially in agriculture, manufacturing, urban management and other fields. It is becoming a key technology to promote intelligent decision-making and automated management.

[0003] Prior art, such as a patent application with announcement number: CN112380704B, discloses a digital twin model correction method and system based on machine vision, the method comprising: designing a digital twin model matching module, which realizes the identification of the target to be corrected, the matching and calling of the digital twin model based on target recognition; designing a digital twin model correction control module based on machine vision, which realizes the formulation and execution control of the digital twin model correction strategy, as well as the version management and update of the digital twin model; designing a digital twin three-dimensional model correction module, which judges whether the digital twin three-dimensional model needs to be corrected based on image recognition and performs correction operations; designing a digital twin motion model correction module, which judges whether the digital twin motion model needs to be corrected based on motion capture and performs correction operations.

[0004] Based on the above scheme, it is found that the limitations of the existing technology include at least the following problems. First, in the process of correcting the digital twin model, the existing technology is limited to correcting only certain specific aspects of the model (such as geometric dimensions or material status), and fails to achieve comprehensive dynamic adjustment and comprehensive analysis of the model, which easily leads to the fact that in actual operation, although the model meets the set standards in theory, it is difficult to accurately map and respond to the health status of the plant in actual application, resulting in limitations in the application of the model and reducing the effectiveness of decision-making and the accuracy of operation. Secondly, the traditional digital twin model correction method usually relies on a large number of manual operations and lacks the necessary automation and intelligent means, which not only leads to low efficiency in the correction process, but also does not respond quickly enough when processing real-time data, greatly limiting the real-time update capability of the model, which is not conducive to real-time monitoring and management of the health status of the plant. In addition, the existing technology also shows deficiencies in the verification and prediction of the model after correction, and fails to effectively use dynamic data analysis and prediction technology. Therefore, it is difficult to predict and adjust possible problems of the plant in the future during the model correction process, resulting in higher maintenance costs and longer adjustment cycles, which in turn affects the timeliness of plant health management and the effectiveness of preventive measures. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method and system for correcting a digital twin model based on machine vision, which solves the problem that the prior art is limited to correcting only certain specific aspects of the model (such as geometric dimensions or material status) during the correction process of the digital twin model, but fails to achieve comprehensive dynamic adjustment and comprehensive analysis of the model, which easily leads to the fact that in actual operation, although the model meets the set standards in theory, it is difficult to accurately map and respond to the health status of the plant in actual application, resulting in limitations in the application of the model, reducing the effectiveness of decision-making and the accuracy of operation. Secondly, the traditional digital twin model correction method usually relies on a large amount of manual operation and lacks the necessary automation and intelligent means, which not only leads to low efficiency in the correction process, but also does not respond quickly enough when processing real-time data, greatly limiting the real-time update capability of the model, and is not conducive to real-time monitoring and management of the health status of the plant. In addition, the prior art also shows deficiencies in verification and prediction after model correction, and fails to effectively use dynamic data analysis and prediction technology. Therefore, it is difficult to predict and adjust possible problems of the plant in the future during the model correction process, resulting in higher maintenance costs and longer adjustment cycles, which in turn affects the timeliness of plant health management and the effectiveness of preventive measures.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a digital twin model correction method based on machine vision, comprising the following steps: real-time acquisition of multispectral image data of several shooting angles of several plants in the experimental planting area to be monitored, and preprocessing; fusing the preprocessed multispectral image data of each shooting angle of each plant in the experimental planting area to be monitored to obtain a three-dimensional voxel data set of each plant in the experimental planting area to be monitored; feature extraction of the three-dimensional voxel data set of each plant in the experimental planting area to be monitored to obtain a plant feature set of each plant in the experimental planting area to be monitored; based on the three-dimensional voxel data set of each plant in the experimental planting area to be monitored, the plant feature set Collect, match and correct the initial digital twin model of the test planting area to be monitored pre-stored in the database to obtain a comprehensive corrected digital twin model of the test planting area to be monitored; based on the comprehensive corrected digital twin model of the test planting area to be monitored, analyze the real-time health index of each plant in the test planting area to be monitored, and conduct a comprehensive analysis to obtain the regional plant real-time health comprehensive index in the test planting area to be monitored; judge and analyze the regional plant real-time health comprehensive index in the test planting area to be monitored and the preset regional plant real-time health comprehensive index interval, and take corresponding measures based on the judgment and analysis results; among them, the specific formula for calculating the regional plant real-time health comprehensive index in the test planting area to be monitored is as follows: Among them, ZhK is the real-time comprehensive health index of regional plants in the experimental planting area to be monitored, and ZjK i is the real-time health index of the i-th plant in the experimental planting area to be monitored, α i is the weighting coefficient of the real-time health index of the i-th plant in the experimental planting area to be monitored, i=1,2,3,…,i0, i0 is the number of plants.

[0007] Furthermore, the multispectral image data specifically includes the pixel value, red light band reflectance, near infrared band reflectance, blue light band reflectance, and multispectral two-dimensional coordinates of each pixel point in the multispectral image; the three-dimensional voxel data set specifically includes the plant comprehensive voxel value, plant comprehensive red light band reflectance, plant comprehensive near infrared band reflectance, plant comprehensive blue light band reflectance, and plant comprehensive three-dimensional coordinate data of each plant voxel point; the plant comprehensive three-dimensional coordinate data includes plant comprehensive three-dimensional horizontal coordinate value, plant comprehensive three-dimensional vertical coordinate value, and plant comprehensive three-dimensional height coordinate value; the plant feature set specifically includes the plant enhanced vegetation index, plant greenness index, plant leaf geometric curvature index of each leaf area, and plant leaf stem geometric curvature index of each leaf stem area.

[0008] Furthermore, the specific steps of obtaining a three-dimensional voxel data set of each plant in the experimental planting area to be monitored are as follows: obtaining the rotation angle of each shooting angle of each plant in the experimental planting area to be monitored and the horizontal focal length value, vertical focal length value, horizontal resolution value, vertical resolution value, physical width value, physical height value of the multispectral sensor and the depth information value between each pixel point of the corresponding shooting device of the multispectral image of each plant in the experimental planting area to be monitored; respectively, the multispectral two-dimensional coordinates of each pixel point in the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored are respectively compared with the horizontal focal length value, vertical focal length value, horizontal resolution value, vertical resolution value, physical width value and physical height value of the multispectral sensor of the corresponding shooting device of the multispectral image of the corresponding shooting angle. A comprehensive analysis is performed on the height values ​​to obtain the three-dimensional horizontal coordinate value and the three-dimensional vertical coordinate value of each pixel point in the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored, and the depth information value between the corresponding shooting device and each pixel point of the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored is marked as the three-dimensional height coordinate value of the corresponding pixel point; a comprehensive analysis is performed on the rotation angle of each shooting angle of each plant in the experimental planting area to be monitored and the three-dimensional horizontal coordinate value, the three-dimensional vertical coordinate value, and the three-dimensional height coordinate value of each pixel point to obtain the shooting three-dimensional horizontal coordinate value, the shooting three-dimensional vertical coordinate value, and the three-dimensional height coordinate value of each pixel point at each shooting angle in a unified three-dimensional coordinate system for each plant in the experimental planting area to be monitored. The three-dimensional vertical coordinate value and the three-dimensional height coordinate value are taken; the three-dimensional horizontal coordinate value, the three-dimensional vertical coordinate value and the three-dimensional height coordinate value of each pixel point of each shooting angle of each plant in the monitored test planting area are compared and analyzed respectively, and the maximum shooting three-dimensional horizontal coordinate value, the maximum shooting three-dimensional vertical coordinate value, the maximum shooting three-dimensional height coordinate value, the minimum shooting three-dimensional horizontal coordinate value, the minimum shooting three-dimensional vertical coordinate value and the minimum shooting three-dimensional height coordinate value of each shooting angle of each plant in the monitored test planting area are obtained, and a comprehensive voxel grid of each plant in the monitored test planting area is established; and the comprehensive voxel grid of each plant in the monitored test planting area is divided based on the preset grid side length, A plurality of voxel grids for each plant in the test planting area to be monitored are obtained; the three-dimensional horizontal coordinate value, the three-dimensional vertical coordinate value, and the three-dimensional height coordinate value of each pixel point at each shooting angle of each plant in the test planting area to be monitored are comprehensively analyzed with the preset grid side length to obtain the row index, column index, and layer index of each pixel point at each shooting angle of each plant in the test planting area to be monitored in the comprehensive voxel grid; and the row index, column index, and layer index of each pixel point at each shooting angle of each plant in the test planting area to be monitored are comprehensively analyzed in combination with statistical methods to obtain a plurality of pixel points of each voxel grid of each plant in the test planting area to be monitored;Combined with the mean value method, the pixel value, three-dimensional horizontal coordinate value, three-dimensional vertical coordinate value, three-dimensional height coordinate value, red light band reflectivity, near infrared band reflectivity, and blue light band reflectivity of each pixel point of each voxel grid of each plant in the monitored test planting area are comprehensively analyzed to obtain the plant comprehensive voxel value, plant comprehensive three-dimensional horizontal coordinate value, plant comprehensive three-dimensional vertical coordinate value, plant comprehensive three-dimensional height coordinate value, plant comprehensive red light band reflectivity, plant comprehensive near infrared band reflectivity, and plant comprehensive blue light band reflectivity of each voxel grid of each plant in the monitored test planting area, that is, the plant comprehensive voxel value, plant comprehensive three-dimensional horizontal coordinate value, plant comprehensive three-dimensional vertical coordinate value, plant comprehensive three-dimensional height coordinate value, plant comprehensive red light band reflectivity, plant comprehensive near infrared band reflectivity, and plant comprehensive blue light band reflectivity of each plant voxel point of each plant in the monitored test planting area, and marked as a three-dimensional voxel data set. ;

[0009] Furthermore, the specific formula for calculating the three-dimensional horizontal coordinate value and the three-dimensional vertical coordinate value of each pixel point in the multispectral image of each shooting angle of each plant in the test planting area to be monitored is as follows: Among them, SpB imn is the three-dimensional horizontal coordinate value of the nth pixel point in the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, XsH imn is the two-dimensional horizontal coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SpF im is the horizontal resolution value of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, WkD im is the physical width value of the multispectral sensor of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SpJ im is the horizontal focal length of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SdZ imn is the three-dimensional height coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored (i.e., the depth information value between the corresponding shooting device and the nth pixel point of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored), CzB imn is the three-dimensional vertical coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored, CsH imn CpF is the two-dimensional vertical coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored. imWgD is the vertical resolution value of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, im is the physical height value of the multispectral sensor of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, CpJ im is the vertical focal length value of the shooting device corresponding to the multispectral image of the mth shooting angle of the ith plant in the experimental planting area to be monitored, i = 1, 2, 3, ..., i0, i0 is the number of plants, m = 1, 2, 3, ..., m0, m0 is the number of shooting angles, and n = 1, 2, 3, ..., n0, n0 is the number of pixels.

[0010] Furthermore, the specific steps for obtaining the plant feature set of each plant in the test planting area to be monitored are as follows: combining the threshold segmentation algorithm to comprehensively analyze the plant comprehensive red light band reflectivity and the plant comprehensive near-infrared band reflectivity of each plant voxel point of each plant in the test planting area to be monitored, and obtaining a number of leaf plant voxel points and a number of leaf stem plant voxel points of each plant in the test planting area to be monitored; and combining the edge detection algorithm to comprehensively analyze the several leaf plant voxel points and the leaf stem plant voxel points of each plant in the test planting area to be monitored, and obtaining a number of leaf areas and a number of leaf stem areas of each plant in the test planting area to be monitored; the plant comprehensive red light band reflectivity and the plant comprehensive near-infrared band reflectivity of each leaf plant voxel point of each leaf area of ​​each plant in the test planting area to be monitored are obtained. The SHOT algorithm is used to comprehensively analyze the three-dimensional coordinates of each leaf plant voxel point in each leaf area of ​​each plant in the monitored test planting area and each leaf stem plant voxel point in each leaf stem area to obtain the plant leaf geometric bending index of each leaf area and the plant leaf stem geometric bending index of each leaf stem area of ​​each plant in the monitored test planting area; based on the plant enhanced vegetation index, plant greenness index, plant leaf geometric bending index of each leaf area of ​​each plant in the monitored test planting area and the plant leaf stem geometric bending index of each leaf stem area, a plant feature set of each plant in the monitored test planting area is established.

[0011] Furthermore, the specific steps of calculating the plant enhanced vegetation index and plant greenness index of each leaf area of ​​each plant in the monitored test planting area are as follows: Among them, ZqB ih is the plant enhanced vegetation index of the hth leaf area of ​​the ith plant in the experimental planting area to be monitored, is the gain factor stored in the database, HgB ihj is the near-infrared band reflectance corresponding to the jth leaf voxel point in the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, JgB ihj is the infrared band reflectivity corresponding to the jth leaf voxel point of the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, θ is the red light band coefficient stored in the database, LgB ihj is the blue light band reflectance corresponding to the jth leaf voxel point in the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, is the blue light band coefficient stored in the database, LdZ ih is the plant greenness index of the hth leaf area of ​​the ith plant in the experimental planting area to be monitored, i = 1, 2, 3, …, i0, i0 is the number of plants, h = 1, 2, 3, …, h0, h0 is the number of leaf areas, j = 1, 2, 3, …, j0, j0 is the number of leaf plant voxel points.

[0012] Furthermore, the specific steps of obtaining a comprehensive corrected digital twin model of the test planting area to be monitored are as follows: combining the coordinate origin in the initial digital twin model of the test planting area to be monitored pre-stored in the database to perform position transformation processing on the plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored, and obtain the transformed plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored; respectively compare and analyze the three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored after the position transformation with each model voxel point of each plant in the initial digital twin model, and obtain the plant comprehensive three-dimensional coordinates of several non-overlapping plant voxel points and several overlapping plant voxel points of each plant in the initial digital twin model of the test planting area to be monitored; and respectively compare and analyze the plant comprehensive voxel value, plant comprehensive red light band reflectivity, plant comprehensive three-dimensional coordinates of each overlapping plant voxel point of each plant in the initial digital twin model of the test planting area to be monitored pre-stored in the database. The comprehensive near-infrared band reflectivity, the comprehensive blue light band reflectivity of the plant, the comprehensive voxel value of the model plant, the comprehensive red light band reflectivity of the model plant, the comprehensive near-infrared band reflectivity of the model plant, and the comprehensive blue light band reflectivity of the model plant are analyzed for differences; and based on the corresponding difference analysis results and the iterative correction method, each overlapping plant voxel of each plant and each non-overlapping plant voxel point in the initial digital twin model of the experimental planting area to be monitored pre-stored in the database are corrected, and the initial corrected digital twin model of the experimental planting area to be monitored is obtained; and the plant enhanced vegetation index, plant greenness index, plant leaf geometric bending index of each leaf area of ​​each plant in the experimental planting area to be monitored and the plant leaf stem geometric bending index of each leaf stem area are input into the corrected digital twin model of the experimental planting area to be monitored for updating, so as to obtain the comprehensive corrected digital twin model of the experimental planting area to be monitored.

[0013] Furthermore, the specific steps for obtaining the transformed plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the experimental planting area to be monitored are: obtaining the origin position coordinates of the three-dimensional coordinate system in the initial digital twin model of the experimental planting area to be monitored pre-stored in the database and the origin position coordinates of the shooting three-dimensional coordinate system (i.e., the shooting three-dimensional coordinate system of the shooting angle directly in front of the plant) corresponding to the plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the experimental planting area to be monitored, and performing difference analysis to obtain the origin difference vector in the initial digital twin model of the experimental planting area to be monitored pre-stored in the database; establishing a translation matrix based on the origin difference vector in the initial digital twin model of the experimental planting area to be monitored pre-stored in the database, and performing position translation processing on the shooting three-dimensional coordinate system to obtain a corrected shooting three-dimensional coordinate system, and performing position correction processing on the plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the experimental planting area to be monitored to obtain the transformed plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the experimental planting area to be monitored.

[0014] Furthermore, the specific steps of taking corresponding measures based on the judgment and analysis results are as follows: if the real-time comprehensive health index of regional plants in the monitored experimental planting area is within the preset range of the real-time comprehensive health index of regional plants, no measures are taken; if the real-time comprehensive health index of regional plants in the monitored experimental planting area is lower than the preset lower limit of the real-time comprehensive health index range of regional plants, a first management suggestion is sent to relevant staff; if the real-time comprehensive health index of regional plants in the monitored experimental planting area is higher than the preset upper limit of the real-time comprehensive health index range of regional plants, a second management suggestion is sent to relevant staff.

[0015] A digital twin model correction system based on machine vision includes: a data acquisition module, a data fusion module, a feature extraction module, a matching correction module, a comprehensive analysis module, and a monitoring management module; the data acquisition module is used to acquire multispectral image data of several shooting angles of several plants in a test planting area to be monitored in real time and perform preprocessing; the data fusion module is used to fuse the preprocessed multispectral image data of each shooting angle of each plant in the test planting area to be monitored to obtain a three-dimensional voxel data set of each plant in the test planting area to be monitored; the feature extraction module is used to perform feature extraction on the three-dimensional voxel data set of each plant in the test planting area to be monitored, respectively, to obtain a plant feature set of each plant in the test planting area to be monitored; the matching correction module is used to obtain a plant feature set of each plant in the test planting area to be monitored; The correction module is used to match and correct the initial digital twin model of the test planting area to be monitored pre-stored in the database based on the three-dimensional voxel data set and plant feature set of each plant in the test planting area to be monitored, so as to obtain a comprehensive corrected digital twin model of the test planting area to be monitored; the comprehensive analysis module is used to analyze the real-time health index of each plant in the test planting area to be monitored based on the comprehensive corrected digital twin model of the test planting area to be monitored, and perform a comprehensive analysis to obtain the real-time comprehensive health index of the regional plants in the test planting area to be monitored; the monitoring and management module is used to judge and analyze the real-time comprehensive health index of the regional plants in the test planting area to be monitored and a preset range of the real-time comprehensive health index of the regional plants, and take corresponding measures based on the judgment and analysis results.

[0016] The present invention has the following beneficial effects:

[0017] (1) The digital twin model correction method based on machine vision can provide more detailed spatial information by preprocessing and fusing multi-angle and multi-spectral image data collected in real time, thereby constructing a more accurate three-dimensional voxel data set and correcting the initial model. This not only enhances the accuracy of the digital twin model, but also more accurately reflects the growth status and spatial distribution of the plant, thereby enabling the model to simulate the real environment more realistically and provide a more reliable basis for planting management.

[0018] (2) The digital twin model correction method based on machine vision acquires multispectral image data from different shooting angles in real time and continuously updates the three-dimensional voxel data set, thereby realizing dynamic correction of the digital twin model. This ensures that the model can be adaptively adjusted over time and remain consistent with the actual planting area, thus avoiding deviations between the initial model and the actual situation, and improving the feasibility and accuracy in long-term use.

[0019] (3) The machine vision-based digital twin model correction method can provide accurate growth status analysis for each plant by extracting features and matching the three-dimensional voxel data set of each plant, and then calculate its health index to identify plant health problems. It can also provide personalized intervention measures based on the corrected digital twin model. When the health index of a plant is lower than the set threshold, the model can remind managers to take timely measures, such as adjusting water and fertilizer supply or conducting pest and disease control, thereby improving the growth environment optimization and yield quality of crops, while reducing unnecessary waste of resources.

[0020] (4) The digital twin model correction system based on machine vision can realize real-time monitoring and precise management of plant growth status by introducing multi-module combined processing. The system can obtain multispectral image data in the planting area in real time, and generate accurate three-dimensional voxel data sets through multi-angle image fusion processing, which can timely reflect the growth changes of plants. The feature extraction and model correction modules ensure that the status information of each plant is always consistent with the digital twin model, avoiding the lag problem existing in traditional models. Finally, the monitoring and management module can respond quickly and take targeted intervention measures through real-time judgment and analysis of the comprehensive health index of plants.

[0021] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a digital twin model correction method based on machine vision in the present invention.

[0023] Figure 2 The present invention is a flowchart of the specific steps of obtaining the transformed plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the monitored test planting area in a digital twin model correction method based on machine vision.

[0024] Figure 3 This is a block diagram of a digital twin model correction system based on machine vision in the present invention. DETAILED DESCRIPTION

[0025] The embodiment of the present application solves the problem that the prior art is limited to correcting only certain specific aspects of the model (such as geometric dimensions or material status) during the correction process of the digital twin model, and fails to achieve comprehensive dynamic adjustment and comprehensive analysis of the model, which easily leads to the fact that in actual operation, although the model meets the set standards in theory, it is difficult to accurately map and respond to the health status of the plant in actual application, resulting in limitations in the application of the model, reducing the effectiveness of decision-making and the accuracy of operation. Secondly, the traditional digital twin model correction method usually relies on a large amount of manual operation and lacks the necessary automation and intelligent means, which not only leads to low efficiency in the correction process, but also does not respond quickly enough when processing real-time data, greatly limiting the real-time update capability of the model, and is not conducive to real-time monitoring and management of the health status of the plant. In addition, the prior art also shows deficiencies in verification and prediction after model correction, and fails to effectively use dynamic data analysis and prediction technology. Therefore, it is difficult to predict and adjust possible problems of the plant in the future during the model correction process, resulting in higher maintenance costs and longer adjustment cycles, which in turn affects the timeliness of plant health management and the effectiveness of preventive measures.

[0026] The overall idea of ​​the problem in the embodiment of this application is as follows:

[0027] First, multispectral image data of each shooting angle of several plants in the experimental planting area to be monitored are acquired in real time, and these image data are preprocessed. Next, the image data at different angles are fused to generate a three-dimensional voxel data set of each plant in the experimental planting area to be monitored, and features are extracted to obtain a plant feature set of each plant in the experimental planting area to be monitored. Then, the three-dimensional voxel data set and plant feature set of each plant in the experimental planting area to be monitored are matched and corrected with the pre-stored initial digital twin model to obtain a comprehensive corrected digital twin model of the experimental planting area to be monitored. Based on the comprehensive corrected digital twin model, the real-time health index of each plant is analyzed. By comprehensively analyzing the health status of each plant, the comprehensive health index of the experimental planting area to be monitored is obtained.

[0028] See also Figure 1, an embodiment of the present invention provides a technical solution: a digital twin model correction method based on machine vision, comprising the following steps: real-time acquisition of multispectral image data (obtained by a multispectral camera) of several shooting angles of several plants in the experimental planting area to be monitored (in this implementation example, five shooting angles are obtained, namely, directly in front of the plant, directly behind, on the left side, on the right side, and directly above the plant), and preprocessing; fusing the preprocessed multispectral image data of each shooting angle of each plant in the experimental planting area to be monitored to obtain a three-dimensional voxel data set of each plant in the experimental planting area to be monitored; feature extraction is performed on the three-dimensional voxel data set of each plant in the experimental planting area to be monitored to obtain a three-dimensional voxel data set of each plant in the experimental planting area to be monitored. Plant feature set; based on the three-dimensional voxel data set and plant feature set of each plant in the experimental planting area to be monitored, the initial digital twin model of the experimental planting area to be monitored pre-stored in the database is matched and corrected to obtain a comprehensive corrected digital twin model of the experimental planting area to be monitored; based on the comprehensive corrected digital twin model of the experimental planting area to be monitored, the real-time plant health index (corresponding to the growth status of the plant) of each plant in the experimental planting area to be monitored is analyzed separately, and a comprehensive analysis is performed to obtain the regional plant real-time health comprehensive index in the experimental planting area to be monitored; the regional plant real-time health comprehensive index in the experimental planting area to be monitored is judged and analyzed with a preset regional plant real-time health comprehensive index interval, and corresponding measures are taken based on the judgment and analysis results;

[0029] The specific formula for calculating the real-time comprehensive health index of regional plants in the test planting area to be monitored is as follows: Among them, ZhK is the real-time comprehensive health index of regional plants in the experimental planting area to be monitored, and ZjK i is the real-time health index of the i-th plant in the experimental planting area to be monitored, α i is the weighting coefficient of the real-time health index of the i-th plant in the experimental planting area to be monitored, i=1,2,3,…,i0, i0 is the number of plants.

[0030] It needs to be explained that α i The specific acquisition process is: obtain the plant health reference index of each plant in the experimental planting area to be monitored under the corresponding growth state, and sum them up to obtain the plant health reference index and value, and perform a proportion analysis on the plant health reference index of the i-th plant in the experimental planting area to be monitored under the corresponding growth state and the plant health reference index and value, and use the proportion analysis as the weighting coefficient.

[0031] The multispectral image data specifically includes the pixel value of each pixel point in the multispectral image, the red light band reflectance (i.e., the percentage of light reflection intensity within the red light band to the incident light intensity), the near-infrared band reflectance (i.e., the percentage of light reflection intensity within the near-infrared band to the incident light intensity), the blue light band reflectance (i.e., the percentage of light reflection intensity within the blue light band to the incident light intensity), and the multispectral two-dimensional coordinates. The three-dimensional voxel data set specifically includes the plant comprehensive voxel value of each plant voxel point, the plant comprehensive red light band reflectance, the plant comprehensive near-infrared band reflectance, the plant comprehensive blue light band reflectance, and the plant comprehensive three-dimensional coordinate data. The plant comprehensive three-dimensional coordinate data includes the plant comprehensive three-dimensional horizontal coordinate value, the plant comprehensive three-dimensional vertical coordinate value, and the plant comprehensive three-dimensional height coordinate value. The plant feature set specifically includes the plant enhanced vegetation index, the plant greenness index, the plant leaf geometric curvature index of each leaf area, and the plant leaf stem geometric curvature index of each leaf stem area.

[0032] The multispectral two-dimensional coordinates take the upper left corner of the image as the origin, and the X and Y axes are the rows and columns of the image respectively.

[0033] Specifically, the specific steps for obtaining a three-dimensional voxel data set of each plant in the experimental planting area to be monitored are as follows: obtaining the rotation angle of each shooting angle of each plant in the experimental planting area to be monitored and the horizontal focal length value, vertical focal length value, horizontal resolution value, vertical resolution value, physical width value, physical height value of the multispectral sensor, and depth information value between each pixel point of the corresponding shooting device (multispectral camera) of the multispectral image; respectively, the multispectral two-dimensional coordinates of each pixel point in the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored are respectively compared with the multispectral two-dimensional coordinates of each pixel point in the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored. A comprehensive analysis is performed on the horizontal focal length value, vertical focal length value, horizontal resolution value, vertical resolution value, physical width value and physical height value of the multispectral sensor to obtain the three-dimensional horizontal coordinate value and three-dimensional vertical coordinate value of each pixel point in the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored, and the depth information value between the corresponding shooting device (multispectral camera) and each pixel point of the multispectral image of each shooting angle of each plant in the experimental planting area to be monitored is marked as the three-dimensional height coordinate value of the corresponding pixel point; the rotation angle of each shooting angle of each plant in the experimental planting area to be monitored and the depth information value between each pixel point are marked as the three-dimensional height coordinate value of the corresponding pixel point. The three-dimensional horizontal coordinate values, three-dimensional vertical coordinate values, and three-dimensional height coordinate values ​​of each plant in the monitored test planting area are comprehensively analyzed to obtain the three-dimensional horizontal coordinate values ​​(i.e., the three-dimensional horizontal coordinate values ​​at each shooting angle in the unified three-dimensional coordinate system), the three-dimensional vertical coordinate values ​​(i.e., the three-dimensional vertical coordinate values ​​at each shooting angle in the unified three-dimensional coordinate system), and the three-dimensional height coordinate values ​​(i.e., the three-dimensional height coordinate values ​​at each shooting angle in the unified three-dimensional coordinate system) of each pixel point at each shooting angle in the unified three-dimensional coordinate system (with the position of the shooting device directly in front as the origin of the unified three-dimensional coordinate system); the statistical method is combined to respectively analyze the three-dimensional horizontal coordinate values, the three-dimensional vertical coordinate values, and the three-dimensional height coordinate values ​​of each pixel point at each shooting angle in the unified three-dimensional coordinate system. The three-dimensional horizontal coordinate value, the three-dimensional vertical coordinate value, and the three-dimensional height coordinate value of each pixel point of each shooting angle of each plant in the test planting area are compared and analyzed to obtain the maximum three-dimensional horizontal coordinate value, the maximum three-dimensional vertical coordinate value, the maximum three-dimensional height coordinate value, the minimum three-dimensional horizontal coordinate value, the minimum three-dimensional vertical coordinate value, and the minimum three-dimensional height coordinate value of each shooting angle of each plant in the test planting area to be monitored, and establish a comprehensive voxel grid of each plant in the test planting area to be monitored; and based on the preset grid side length (in this implementation example, the preset grid side length is 0.5mm) is divided into a comprehensive voxel grid for each plant in the test planting area to be monitored, and a plurality of voxel grids for each plant in the test planting area to be monitored are obtained; the shooting three-dimensional horizontal coordinate value, the shooting three-dimensional vertical coordinate value, and the shooting three-dimensional height coordinate value of each pixel point of each shooting angle of each plant in the test planting area to be monitored are respectively comprehensively analyzed with the preset grid side length to obtain the row index (along the X-axis direction, that is, the position in the horizontal direction), column index (along the Y-axis direction, that is, the position in the vertical direction), and layer index (that is, the position in the depth direction) of each pixel point of each shooting angle of each plant in the test planting area to be monitored in the comprehensive voxel grid; and the row index, column index, and layer index of each pixel point of each shooting angle of each plant in the test planting area to be monitored are comprehensively analyzed in combination with the statistical method to obtain a plurality of voxel grids for each plant in the test planting area to be monitored. Pixel point; Combined with the mean value method, the pixel value, three-dimensional horizontal coordinate value, three-dimensional vertical coordinate value, three-dimensional height coordinate value, red light band reflectivity, near infrared band reflectivity, and blue light band reflectivity of each pixel point of each voxel grid of each plant in the monitored test planting area are comprehensively analyzed to obtain the plant comprehensive voxel value, plant comprehensive three-dimensional horizontal coordinate value, plant comprehensive three-dimensional vertical coordinate value, plant comprehensive three-dimensional height coordinate value, plant comprehensive red light band reflectivity, plant comprehensive near infrared band reflectivity, and plant comprehensive blue light band reflectivity of each voxel grid of each plant in the monitored test planting area, that is, the plant comprehensive voxel value, plant comprehensive three-dimensional horizontal coordinate value, plant comprehensive three-dimensional vertical coordinate value, plant comprehensive three-dimensional height coordinate value, plant comprehensive red light band reflectivity, plant comprehensive near infrared band reflectivity, and plant comprehensive blue light band reflectivity of each plant voxel point of each plant in the monitored test planting area, and mark them as three-dimensional voxel data sets. .

[0034] The rotation angle is obtained through an inertial measurement unit.

[0035] The horizontal focal length value, the vertical focal length value, the horizontal resolution value, the vertical resolution value, the physical width value of the sensor, and the physical height value of the sensor are all directly acquired through the device control software, and the sensor is a multispectral sensor in the multispectral camera.

[0036] The depth information value is obtained by the LiDAR sensor.

[0037] The calculation formula for the row index in the integrated voxel grid and the plant integrated voxel value of each plant voxel point at each shooting angle of each plant in the monitored test planting area is as follows: Among them, HsY imnis the row index of the nth pixel in the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SpB imn is the three-dimensional horizontal coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored, ZxS im is the minimum three-dimensional vertical coordinate value of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, TsZ if is the plant comprehensive voxel value of the f-th plant voxel point of the i-th plant in the experimental planting area to be monitored, XsZ ifb The pixel value of the bth pixel point of the fth plant voxel point of the ith plant in the experimental planting area to be monitored, i = 1, 2, 3, …, i0, i0 is the number of plants, m = 1, 2, 3, …, m0, m0 is the number of shooting angles, n = 1, 2, 3, …, n0, n0 is the number of pixels, f = 1, 2, 3, …, f0, f0 is the number of plant voxel points, b = 1, 2, 3, …, b0, b0 is the number of pixels in the plant voxel point.

[0038] It should be explained that the calculation logic of the column index, layer index and row index in the integrated voxel grid for each pixel point at each shooting angle of each plant in the experimental planting area to be monitored is consistent, and the calculation formula is also consistent.

[0039] The calculation logic of the plant's comprehensive three-dimensional horizontal coordinate value, plant's comprehensive three-dimensional vertical coordinate value, plant's comprehensive three-dimensional height coordinate value, plant's comprehensive red light band reflectivity, plant's comprehensive near-infrared band reflectivity, and plant's comprehensive blue light band reflectivity of each voxel grid of each plant in the experimental planting area to be monitored is consistent with the plant's comprehensive voxel value, and the calculation formula is also consistent.

[0040] In this embodiment, each shooting angle is directly in front of the plant, directly behind, on the left, on the right, and directly above, and the front of the test planting area to be monitored is set as the rotation reference direction. The rotation angles corresponding to each shooting angle are 0°, 180° (rotation 180° around the Y axis), 90° (rotation 90° around the Z axis), -90° (rotation -90° around the Z axis), and 90° (rotation 90° around the X axis).

[0041] The specific formula for calculating the three-dimensional horizontal coordinate value and the three-dimensional vertical coordinate value of each pixel point in the multispectral image of each shooting angle of each plant in the test planting area to be monitored is as follows: Among them, SpB imn is the three-dimensional horizontal coordinate value of the nth pixel point in the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, XsH imnis the two-dimensional horizontal coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SpF im is the horizontal resolution value of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, WkD im is the physical width value of the multispectral sensor of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SpJ im is the horizontal focal length of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, SdZ imn is the three-dimensional height coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored (i.e., the depth information value between the corresponding shooting device and the nth pixel point of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored), CzB imn is the three-dimensional vertical coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored, CsH imn CpF is the two-dimensional vertical coordinate value of the nth pixel point at the mth shooting angle of the i-th plant in the experimental planting area to be monitored. im WgD is the vertical resolution value of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, im is the physical height value of the multispectral sensor of the corresponding shooting device of the multispectral image of the mth shooting angle of the i-th plant in the experimental planting area to be monitored, CpJ im is the vertical focal length value of the shooting device corresponding to the multispectral image of the mth shooting angle of the ith plant in the experimental planting area to be monitored, i = 1, 2, 3, ..., i0, i0 is the number of plants, m = 1, 2, 3, ..., m0, m0 is the number of shooting angles, and n = 1, 2, 3, ..., n0, n0 is the number of pixels.

[0042] In this implementation scheme, by comprehensively analyzing the multispectral images collected from multiple shooting angles (such as the front, the back, the left, the right, and the top) of each plant in the monitored test planting area, the three-dimensional information of the plant can be obtained from multiple dimensions, thereby effectively avoiding the information loss and errors that may exist under a single perspective, making the acquired three-dimensional voxel data set more accurate and comprehensive. Secondly, the multispectral data of each pixel point is accurately calculated and fused with its three-dimensional coordinate value (horizontal, vertical, and depth) to describe the spatial distribution of the plant and its growth state, thereby making the three-dimensional modeling of the plant more detailed and able to accurately capture the changes of the plant at different growth stages and under different environmental conditions. In addition, the three-dimensional voxel data set of each plant obtained by multi-angle and multi-parameter fusion and analysis in this step can be used to correct the existing digital twin model, thereby providing a real and detailed basis for the correction process, making the corrected digital twin model more in line with the actual situation, thereby improving the prediction accuracy and decision-making effect of the model. Finally, by integrating accurate three-dimensional voxel data with multispectral image data, the recognition accuracy of the health status of the plant is improved, and a basis for the optimal allocation of agricultural resources can also be provided.

[0043] Specifically, the specific steps for obtaining the plant feature set of each plant in the test planting area to be monitored are as follows: combining the threshold segmentation algorithm to comprehensively analyze the plant comprehensive red light band reflectivity and the plant comprehensive near-infrared band reflectivity of each plant voxel point of each plant in the test planting area to be monitored, and obtaining a number of leaf plant voxel points and a number of leaf stem plant voxel points of each plant in the test planting area to be monitored; and combining the edge detection algorithm to comprehensively analyze the several leaf plant voxel points and the leaf stem plant voxel points of each plant in the test planting area to be monitored, and obtaining a number of leaf areas and a number of leaf stem areas of each plant in the test planting area to be monitored; the plant comprehensive red light band reflectivity, the plant comprehensive near-infrared band reflectivity, the plant comprehensive blue light band reflectivity of each leaf plant voxel point of each leaf area of ​​each plant in the test planting area to be monitored are obtained. A comprehensive analysis is performed on the reflectance of the light band to obtain the plant enhanced vegetation index (plant growth status) and plant greenness index (chlorophyll content) of each leaf area of ​​each plant in the experimental planting area to be monitored; the three-dimensional coordinates of each leaf plant voxel point in each leaf area of ​​each plant in the experimental planting area to be monitored and each leaf stem plant voxel point in each leaf stem area are comprehensively analyzed in combination with the SHOT algorithm to obtain the plant leaf geometric bending index of each leaf area of ​​each plant in the experimental planting area to be monitored and the plant leaf stem geometric bending index of each leaf stem area; based on the plant enhanced vegetation index of each leaf area of ​​each plant in the experimental planting area to be monitored, the plant greenness index, the plant leaf geometric bending index and the plant leaf stem geometric bending index of each leaf stem area, a plant feature set of each plant in the experimental planting area to be monitored is established.

[0044] Among them, the specific steps of obtaining a number of leaf plant voxel points and a number of leaf stem plant voxel points of each plant in the test planting area to be monitored are: analyzing a number of leaf plant voxel points and leaf stem plant voxel points of each plant in the test planting area to be monitored, obtaining the normalized vegetation index corresponding to each plant voxel point of each plant in the test planting area to be monitored, and comparing and analyzing with the preset normalized vegetation index threshold respectively; if the normalized vegetation index of the plant voxel point of each plant in the test planting area to be monitored is higher than the normalized vegetation index threshold, then the plant voxel point is a leaf plant voxel point; if the normalized vegetation index of the plant voxel point of each plant in the test planting area to be monitored is lower than the normalized vegetation index threshold, then the plant voxel point is a leaf stem plant voxel point; wherein, the specific formula for calculating the normalized vegetation index corresponding to each plant voxel point of each plant in the test planting area to be monitored is as follows: Among them, GyZ in is the normalized vegetation index corresponding to the nth plant voxel point of the ith plant in the experimental planting area to be monitored, JgB inis the near-infrared band reflectance corresponding to the nth plant voxel point of the ith plant in the experimental planting area to be monitored, HgB in is the red light band reflectivity corresponding to the nth plant voxel point of the i-th plant in the experimental planting area to be monitored, i = 1, 2, 3, ..., i0, i0 is the number of plants, n = 1, 2, 3, ..., n0, n0 is the number of plant voxel points.

[0045] In this implementation example, the edge detection algorithm is the Canny algorithm, and the specific process of obtaining several leaf areas and leaf stem areas of each plant in the test planting area to be monitored is: using the three-dimensional Canny algorithm, the gradient is calculated in three directions (x, y, z) to obtain a three-dimensional gradient vector. For each voxel, its gradient strength and direction are calculated, and the gradient strength can be used to determine whether the plant voxel point is on the boundary, and the direction can help us identify the change trend of the boundary in the three-dimensional space. The plant voxel point with a larger gradient is taken, and the edge with the highest intensity is retained to accurately detect the boundary. The gradient intensity is non-maximum suppressed. For each plant voxel point, the intensity of its adjacent plant voxel points is checked along the gradient direction. If the intensity of the current voxel is not the local maximum, it is suppressed (set to 0) , thereby reducing the interference of false edges and obtaining more detailed edges. Two thresholds (high threshold and low threshold) are set to screen edges. Voxels above the high threshold are marked as strong edges, and voxels below the low threshold are suppressed. Voxels between the two will be retained or suppressed according to the connection relationship with the strong edge to ensure the continuity of the edge. Starting from the strong edge voxels, edge tracing is performed, and the connected edge voxels are classified into the same area. All connected edge voxel areas are marked to distinguish the leaf and leaf stem areas. The edge plant voxel points are connected to form a closed voxel boundary, so as to obtain the complete area of ​​each leaf and leaf stem. Using the obtained boundary voxel information, the voxel area is divided into two main parts: leaf and leaf stem. The edge of the leaf is generally more detailed and extended, while the leaf stem area has a more concentrated boundary structure.

[0046] The specific process of obtaining the plant leaf geometric bending index of each leaf area in the depth image of each plant in the experimental planting area to be monitored is: select any number of leaf plant voxel points in each leaf area in the depth image of each plant in the experimental planting area to be monitored as target plant voxel points, that is, several target plant voxel points, and select other plant voxel points within a certain range of the corresponding target plant voxel points as neighborhoods, that is, several neighborhood leaf plant voxel points in several neighborhoods, and analyze the target plant voxel points of each leaf area in the depth image of each plant in the selected experimental planting area to be monitored with the comprehensive three-dimensional spatial coordinates of each neighborhood leaf plant voxel point in each neighborhood, so as to obtain the geometric bending index of each leaf area in the depth image of each plant in the experimental planting area to be monitored.

[0047] The specific formula for calculating the plant leaf geometric bending index of each leaf area of ​​each plant in the test planting area to be monitored is as follows: Among them, SjH ih is the plant leaf geometric bending index of the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, Zk ihj is the Z-axis coordinate value of the j-th target plant voxel point in the h-th leaf area of ​​the i-th plant in the experimental planting area to be monitored, Zk ihgs is the Z-axis coordinate value of the voxel point of the leaf plant in the sth neighborhood of the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, ihj is the X-axis coordinate value of the j-th target plant voxel point in the h-th leaf area of ​​the i-th plant in the experimental planting area to be monitored, Xk ihgs Yk is the X-axis coordinate value of the voxel point of the sth neighboring leaf plant in the gth neighborhood of the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, ihj Yk is the Y-axis coordinate value of the j-th target plant voxel point in the h-th leaf area of ​​the i-th plant in the experimental planting area to be monitored, ihgs is the Y-axis coordinate value of the sth neighboring leaf plant voxel point in the gth neighborhood of the hth leaf region of the ith plant in the monitored test planting area, i=1,2,3,…,i0,i0 is the number of plants, h=1,2,3,…,h0,h0 is the number of leaf regions, j=1,2,3,…,j0,j0 is the number of target plant voxel points, g=1,2,3,…,g0,g0 is the number of neighborhoods, and s=1,2,3,…,s0,s0 is the number of neighborhood leaf plant voxel points.

[0048] The plant leaf stem geometric bending index of each leaf stem area in the depth image of each plant in the test planting area to be monitored is consistent with the plant leaf stem geometric bending index of each leaf stem area in the steps of obtaining the same, and the calculation logic and calculation formula are consistent.

[0049] In this implementation, by combining multiple algorithms (such as threshold segmentation, edge detection and geometric curvature index analysis), the leaf and stem areas of each plant and the corresponding features are accurately extracted, thereby reducing the errors that may be caused by the traditional single algorithm, ensuring that the growth characteristic data of each plant is more accurate, and then capturing the specific form of the plant in space, thereby providing more detailed information for farm management, which is of great significance for refined agricultural operations, crop growth monitoring, pest and disease control, etc., and can more accurately judge the health and growth of each plant. Secondly, combined with the reflectance data of the red, near-infrared and blue light bands, as well as the plant greenness index and enhanced vegetation index Indicators such as the growth status and geometric bending index of each plant can be used to evaluate the health status of the plant from multiple angles to reveal the nutritional status of the plant. By accurately analyzing the growth status and geometric bending index of each plant, agricultural producers can more reasonably plan the use of water, fertilizer and other resources. In addition, combined with the SHO algorithm and three-dimensional geometric analysis, it can provide more accurate target areas for automated agricultural equipment (such as drones and automated harvesters). Finally, by obtaining the growth characteristics and health status of each plant in real time, agricultural producers can make quick adjustments to solve immediate problems. For example, when the greenness of leaves in a certain area decreases, corresponding water and fertilizer adjustment suggestions can be provided immediately to avoid potential growth crises.

[0050] Specifically, the specific steps for calculating the plant enhanced vegetation index and plant greenness index of each leaf area of ​​each plant in the test planting area to be monitored are as follows: Among them, ZqB ih is the plant enhanced vegetation index of the hth leaf area of ​​the ith plant in the experimental planting area to be monitored, is the gain factor stored in the database. In this implementation example, the value is 2.5. ihj is the near-infrared band reflectance corresponding to the jth leaf voxel point in the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, JgB ihj is the infrared band reflectivity corresponding to the jth leaf voxel point in the hth leaf area of ​​the i-th plant in the test planting area to be monitored, θ is the red light band coefficient stored in the database, and in this implementation example, the value is 6, LgB ihj is the blue light band reflectance corresponding to the jth leaf voxel point in the hth leaf area of ​​the i-th plant in the experimental planting area to be monitored, is the blue light band coefficient stored in the database, and in this implementation example, the value is 7.5, LdZih is the plant greenness index of the hth leaf area of ​​the ith plant in the experimental planting area to be monitored, i = 1, 2, 3, …, i0, i0 is the number of plants, h = 1, 2, 3, …, h0, h0 is the number of leaf areas, j = 1, 2, 3, …, j0, j0 is the number of leaf plant voxel points.

[0051] The specific implementation example of calculating the plant enhanced vegetation index and plant greenness index of a leaf area of ​​a certain plant in the test planting area to be monitored is as follows, and the following data are available:

[0052] There are 20 plant voxel points in a leaf area of ​​a plant in the test planting area to be monitored.

[0053] The reflectance of the red light band of the 20 plant voxel points is: 0.18, 0.22, 0.15, 0.24, 0.20, 0.19, 0.23, 0.21, 0.25, 0.16, 0.17, 0.19, 0.18, 0.20, 0.22, 0.23, 0.24, 0.21, 0.15, 0.20;

[0054] The near-infrared reflectance of the 20 plant voxel points is: 0.58, 0.65, 0.60, 0.62, 0.68, 0.64, 0.66, 0.70, 0.67, 0.61, 0.59, 0.65, 0.63, 0.62, 0.60, 0.64, 0.66, 0.67, 0.69, 0.68;

[0055] The reflectances of the blue light band of the 20 plant voxel points are: 0.14, 0.18, 0.12, 0.16, 0.13, 0.17, 0.19, 0.15, 0.14, 0.16, 0.12, 0.18, 0.17, 0.15, 0.14, 0.19, 0.13, 0.16, 0.14, 0.18;

[0056] Then the plant enhanced vegetation index of a leaf area of ​​a certain plant in the monitored test planting area = ((2.5*(0.58-0.18) / (0.58+(6*0.18+7.5*0.14)))+(2.5*(0.65-0.22) / (0.65+(6*0.22+7.5*0.18)))+(2.5*(0.60-0.15) / (0.60+(6*0.15+7.5*0.12)))+(2.5*(0.62-0.24) / (0.62+(6*0.24+7.5*0.16)))+(2.5*(0.68-0.20) / (0 .68+(6*0.20+7.5*0.13)))+(2.5*(0.64-0.19) / (0.64+(6*0.19+7.5*0.17)))+(2.5*(0.66-0.23) / (0.66+(6*0.18+7.5*0.14)))+(2.5*(0.70-0.21) / (0.70+(6*0.21+7.5*0.15)))+(2.5*(0.67-0.25) / (0.67+(6*0.25+7.5*0.14)))+(2.5*(0.61-0.17) / (0.61+(6*0.17 +7.5*0.16)))+(2.5*(0.59-0.17) / (0.59+(6*0.17+7.5*0.12)))+(2.5*(0.65-0.19) / (0.65+(6*0.19+7.5*0.18)))+(2.5*(0.63-0.18) / (0.63+(6*0.18+7.5*0.17)))+(2.5*(0.62-0.20) / (0.62+(6*0.20+7.5*0.15)))+(2.5*(0.60-0.22) / (0.60+(6*0.22+7.5*0.14)) )+(2.5*(0.64-0.23) / (0.64+(6*0.23+7.5*0.19)))+(2.5*(0.66-0.24) / (0.66+(6*0.24+7.5*0.13)))+(2.5*(0.67-0.21) / (0.67+(6*0.21+7.5*0.1+)))+(2.5*(0.69-0.15) / (0.69+(6*0.15+7.5*0.14)))+(2.5*(0.68-0.20) / (0.68+(6*0.20+7.5*0.18)))) / 20≈1.77.

[0057] Then the plant greenness index of a leaf area of ​​a plant in the monitored test planting area = ln|1+((0.58 / (0.18+1)+(0.65 / (0.22+1))+(0.60 / (0.15+1))+(0.62 / (0.24+1))+(0.68 / (0.20+1))+(0.64 / (0.19+1))+(0.66 / (0.23+1))+(0.70 / (0.21+1))+(0.67 / (0.25+1))++(0. 61 / (0.16+1))+(0.59 / (0.17+1))+(0.65 / (0.19+1))+(0.63 / (0.18+1))+(0.62 / (0.20+1))+(0.60 / (0.22+1))+(0.64 / (0.23+1))+(0.66 / (0.24+1))+(0.67 / (0.21+1))+(0.69 / (0.15+1))+(0.68 / (0.20+1)) / 20)|≈0.627.

[0058] In this implementation scheme, the enhanced vegetation index and plant greenness index of the plant are calculated to evaluate the growth status, photosynthesis efficiency and chlorophyll content of the plant, which helps to timely discover plant growth problems, such as a decrease in photosynthetic capacity or insufficient nutrients, so as to take corresponding intervention measures. Secondly, the enhanced vegetation index and greenness index provide a more accurate plant health assessment method than the traditional vegetation index, which can help researchers monitor the growth dynamics of the plant, discover potential pests and diseases or environmental stresses at an early stage, and take effective management strategies in advance. At the same time, through detailed analysis of each plant and each leaf area, the different growth characteristics of the plant can be accurately monitored at the regional level, thereby providing data support for agricultural management such as precision fertilization, irrigation and pest and disease control. Finally, based on real-time data analysis, the changing trends of plants at different growth stages can be captured, thereby achieving more detailed spatiotemporal data monitoring.

[0059] Specifically, the specific steps for obtaining the comprehensive modified digital twin model of the test planting area to be monitored are as follows: combining the coordinate origin in the initial digital twin model of the test planting area to be monitored pre-stored in the database to perform position transformation processing on the plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored (in this implementation example, the three-dimensional coordinates in the initial digital twin model are consistent with the scale size of the plant comprehensive three-dimensional coordinates, and the initial digital twin model only contains model voxel points related to the plant body), and obtain the transformed each plant voxel point of each plant in the test planting area to be monitored. The plant comprehensive three-dimensional coordinates of the voxel points (in the three-dimensional coordinate system of the initial digital twin model) of each plant in the test planting area to be monitored are compared and analyzed respectively with each model voxel point of each plant in the initial digital twin model to obtain the plant comprehensive three-dimensional coordinates of several non-overlapping plant voxel points (i.e., removing the overlapping parts of the plant voxel points and the model voxel points) and the plant comprehensive three-dimensional coordinates of several overlapping plant voxel points of each plant in the initial digital twin model in the test planting area to be monitored; and the plant comprehensive three-dimensional coordinates of the plant voxel points pre-stored in the database are compared and analyzed respectively. The plant comprehensive voxel value, plant comprehensive red light band reflectivity, plant comprehensive near-infrared band reflectivity, plant comprehensive blue light band reflectivity of each overlapping plant voxel point of each plant in the initial digital twin model of the test planting area to be monitored are analyzed for differences with the model plant comprehensive voxel value, model plant comprehensive red light band reflectivity, model plant comprehensive near-infrared band reflectivity, and model plant comprehensive blue light band reflectivity; and based on the corresponding difference analysis results and the iterative correction method, each overlapping plant voxel point and each non-overlapping plant voxel point of each plant in the initial digital twin model of the test planting area to be monitored pre-stored in the database are corrected. The model plant comprehensive voxel value of the plant voxel point, the model plant comprehensive red light band reflectivity, the model plant comprehensive near-infrared band reflectivity, and the model plant comprehensive blue light band reflectivity are corrected to obtain the initial corrected digital twin model of the experimental planting area to be monitored; and the plant enhanced vegetation index, plant greenness index, plant leaf geometric bending index of each leaf area of ​​each plant in the experimental planting area to be monitored and the plant leaf stem geometric bending index of each leaf stem area are input into the corrected digital twin model of the experimental planting area to be monitored for updating to obtain the comprehensive corrected digital twin model of the experimental planting area to be monitored.

[0060] Among them, the iterative correction method is to use the differences identified in the difference analysis as the initial correction value. The differences are mainly reflected in the comprehensive value of the voxel point and the reflectivity value of each band. The initial difference value can be directly used to preliminarily correct the model voxel data. The initial correction value is applied to the initial model, and the red light, near-infrared, and blue light band reflectivity and the comprehensive voxel value of the model voxel are updated. The preliminary corrected model is compared with the actual measured data, and the error is calculated. Based on the calculated residual or error, the correction value is adjusted, and the new correction value continues to be applied to the model. The error is recalculated after each iteration until the error is less than the predetermined threshold.

[0061] Among them, the specific process of the update processing is: obtain the initial plant enhanced vegetation index, initial plant greenness index, initial plant leaf geometric bending index of each leaf area in the initial digital twin model of the test planting area to be monitored, and the initial plant leaf stem geometric bending index of each leaf stem area of ​​each plant in the test planting area to be monitored is pre-stored in the database, and the plant enhanced vegetation index, plant greenness index, plant leaf geometric bending index of each leaf area and the plant leaf stem geometric bending index of each leaf stem area are replaced with the initial plant enhanced vegetation index, initial plant greenness index, initial plant leaf geometric bending index of each leaf area and the initial plant leaf stem geometric bending index of each leaf stem area in the initial digital twin model of the test planting area to be monitored pre-stored in the database.

[0062] In this implementation scheme, through the position transformation processing and difference analysis of the plant voxel points, the deviations in the model can be detected and corrected in real time, so that the initial model is closer to the state of the actual planting area, thereby improving the accuracy of the model's reflection of the real environment. The accurate three-dimensional coordinate and reflectivity data correction can effectively supplement the gaps and errors in the model and enhance the credibility of the model. Secondly, the model is dynamically updated through the iterative correction method, which avoids frequent manual intervention and adjustment, thereby improving the adaptability of the model and providing more timely and accurate feedback on the growth and changes of plants in a dynamic environment. In addition, the corrected digital twin model not only reflects the three-dimensional spatial distribution of the plant, but also integrates reflectivity data (such as red light, near-infrared and blue light band reflectivity), so as to identify the health status and growth changes of the plant, so that researchers can accurately Resource allocation, pest and disease monitoring, and adaptation measures to environmental changes can be carried out in an appropriate manner, thereby improving the controllability and efficiency of crop production. The revised comprehensive digital twin model incorporates the analysis of all relevant geometric characteristics of leaves and stems (such as the bending index), making the growth characteristics of each plant more comprehensive and enabling better health status assessments, thereby promptly identifying problems in plant growth, such as leaf water shortage or other growth obstacles, and taking targeted measures. Finally, combining the initial model in the database with real-time monitoring data, it can monitor the dynamic state of the plant in all directions in space and time. Through the correction of plant voxel points, difference analysis, and enhanced vegetation index and other data input, the digital twin model can fully integrate the multi-dimensional data of the environment and plants, providing richer information for subsequent plant growth prediction and decision analysis.

[0063] Specifically, Figure 2As shown in the figure, the specific steps for obtaining the transformed plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored are: obtaining the origin position coordinates of the three-dimensional coordinate system in the initial digital twin model of the test planting area to be monitored pre-stored in the database and the origin position coordinates of the shooting three-dimensional coordinate system (that is, the shooting three-dimensional coordinate system of the shooting angle directly in front of the plant) corresponding to the plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored, and performing difference analysis to obtain the origin difference in the initial digital twin model of the test planting area to be monitored pre-stored in the database. A value vector is obtained (i.e., the horizontal coordinate difference, vertical coordinate difference, and height coordinate difference between the coordinate origin position of the three-dimensional coordinate system and the photographed three-dimensional coordinate system); a translation matrix is ​​established based on the origin difference vector in the initial digital twin model of the test planting area to be monitored which is pre-stored in the database, and the photographed three-dimensional coordinate system is positionally translated to obtain a corrected photographed three-dimensional coordinate system, and the plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored are positionally corrected to obtain the transformed plant comprehensive three-dimensional coordinates of each plant voxel point of each plant in the test planting area to be monitored.

[0064] In this implementation scheme, the coordinate system of the digital twin model pre-stored in the database is aligned with the coordinate system of the actual shooting device through difference analysis and the establishment of a translation matrix, ensuring the spatial consistency between the model data and the actual environment, thereby improving the spatial positioning accuracy of the data, reducing the model deviation caused by coordinate errors, and thus improving the reliability of the model. Secondly, real-time coordinate updates at different shooting device positions and times are realized, so that the digital twin model can dynamically reflect the actual changes of the shooting device and the monitoring area, thereby ensuring that the real-time data can accurately reflect the actual situation. In addition, by connecting the three-dimensional coordinates of the shooting device with the coordinate system of the digital twin model, data from different sources (such as satellite images, drone photography, ground sensors, etc.) can be effectively integrated, thereby providing more comprehensive and detailed ground or plant information, thereby supporting more accurate agricultural monitoring and management. Finally, by obtaining a unified three-dimensional coordinate system, different data (such as the reflectivity of plant voxel points, health index, etc.) can be processed and analyzed in the same coordinate system, thereby simplifying the subsequent analysis and comparison of the data, and providing a clearer and more consistent basis for modeling, prediction and decision-making.

[0065] Specifically, the specific steps of taking corresponding measures based on the judgment and analysis results are as follows: if the regional plant real-time health comprehensive index in the monitored test planting area is within the preset regional plant real-time health comprehensive index range, no measures are taken; if the regional plant real-time health comprehensive index in the monitored test planting area is lower than the preset lower limit of the regional plant real-time health comprehensive index range (that is, the minimum value in the regional plant real-time health comprehensive index range), a first management suggestion is sent to relevant staff, that is, providing maintenance measures (such as adding an appropriate amount of fertilizer or nutrients, adjusting water supply, or optimizing light conditions, targeting the specific reasons for the decline in health index, such as lack of water, lack of nutrition, diseases, etc., and take corresponding improvement measures) and increase the monitoring frequency (to ensure that the maintenance measures are effective and avoid a further decline in the health index); if the regional plant real-time health comprehensive index in the monitored test planting area is higher than the preset upper limit of the regional plant real-time health comprehensive index range (that is, the maximum value in the regional plant real-time health comprehensive index range), a second management suggestion will be sent to the relevant staff, that is, to provide suggestions on optimizing resource supply (avoiding adverse plant reactions caused by excessive fertilization, watering or light) and preventing excessive plant growth (adjusting the management plan to maintain an appropriate growth rate and health status, and preventing problems such as structural instability caused by rapid growth of plants).

[0066] In this implementation scheme, by real-time monitoring of the comprehensive health index of the plants in the test planting area and taking corresponding measures according to the preset health interval, the root cause of the plant health problem can be accurately located, so that the management strategy can be adjusted according to specific needs, such as increasing water supply when there is a lack of water, supplementing fertilizers when there is a lack of nutrition, etc., thereby achieving targeted management and avoiding the inefficiency of blind policy implementation. Secondly, real-time monitoring of the comprehensive health index and taking measures according to the deviation of the index from the preset interval can intervene in the early stage of the problem to avoid further deterioration of the problem. For example, when the health index is lower than the preset lower limit, by increasing the monitoring frequency and taking maintenance measures, the plant can be prevented from suffering a large loss due to malnutrition or lack of water. This timeliness reduces the risk of agricultural production and improves the stability of yield. Finally, by maintaining an appropriate growth rate and health state, the structural instability caused by rapid growth or excess resources of plants can be avoided, which will then help the continuous and healthy growth of plants and improve the resistance of plants to environmental changes (such as climate change, pests and diseases, etc.).

[0067] See also Figure 3The embodiment of the present invention provides a technical solution: a digital twin model correction system based on machine vision, including a data acquisition module, a data fusion module, a feature extraction module, a matching correction module, a comprehensive analysis module, and a monitoring management module; the data acquisition module is used to acquire multispectral image data of several shooting angles of several plants in the experimental planting area to be monitored in real time, and perform preprocessing; the data fusion module is used to fuse the preprocessed multispectral image data of each shooting angle of each plant in the experimental planting area to be monitored, so as to obtain a three-dimensional voxel data set of each plant in the experimental planting area to be monitored; the feature extraction module is used to perform feature extraction on the three-dimensional voxel data set of each plant in the experimental planting area to be monitored, so as to obtain a plant data set of each plant in the experimental planting area to be monitored. feature set; a matching and correction module, which is used to match and correct the initial digital twin model of the test planting area to be monitored pre-stored in the database based on the three-dimensional voxel data set and plant feature set of each plant in the test planting area to be monitored, so as to obtain a comprehensive corrected digital twin model of the test planting area to be monitored; a comprehensive analysis module, which is used to analyze the real-time health index of each plant in the test planting area to be monitored based on the comprehensive corrected digital twin model of the test planting area to be monitored, and perform a comprehensive analysis to obtain the real-time comprehensive health index of the regional plants in the test planting area to be monitored; a monitoring and management module, which is used to judge and analyze the real-time comprehensive health index of the regional plants in the test planting area to be monitored with a preset range of the real-time comprehensive health index of the regional plants, and take corresponding measures based on the judgment and analysis results.

[0068] In summary, this application has at least the following effects:

[0069] Preprocessing and fusing multi-angle and multi-spectral image data collected in real time can provide more detailed spatial information, thereby constructing a more accurate three-dimensional voxel data set and correcting the initial model. This not only enhances the accuracy of the digital twin model, but also more accurately reflects the growth status and spatial distribution of the plants, allowing the model to more realistically simulate the real environment and provide a more reliable basis for planting management.

[0070] By acquiring multispectral image data from different shooting angles in real time and continuously updating the three-dimensional voxel data set, dynamic correction of the digital twin model can be achieved, thereby ensuring that the model can adaptively adjust over time and remain consistent with the actual planting area, avoiding deviations between the initial model and the actual situation, and thereby improving the feasibility and accuracy in long-term use.

[0071] By extracting features and matching the three-dimensional voxel data set of each plant, it is possible to provide accurate growth status analysis for each plant, and then calculate its health index to identify plant health problems. It is also possible to provide personalized intervention measures based on the corrected digital twin model. When the health index of a plant is lower than the set threshold, the model can remind managers to take timely measures, such as adjusting water and fertilizer supply or conducting disease and pest control, thereby improving the growth environment optimization and yield quality of crops, while reducing unnecessary waste of resources.

[0072] By introducing multi-module combined processing, real-time monitoring and precise management of plant growth status can be achieved. The system can obtain multispectral image data in the planting area in real time, and generate accurate three-dimensional voxel data sets through multi-angle image fusion processing, which can timely reflect the growth changes of plants. The feature extraction and model correction modules ensure that the status information of each plant is always consistent with the digital twin model, avoiding the lag problem existing in traditional models. Finally, the monitoring and management module can respond quickly and take targeted intervention measures through real-time judgment and analysis of the comprehensive health index of plants.

[0073] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0074] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A digital twin model correction method based on machine vision, characterized in that: The following steps are involved: Real-time acquisition of multispectral image data of several plants in the experimental planting area to be monitored at several shooting angles, and pre-processing; The pre-processed multispectral image data of each shooting angle of each plant in the test planting area to be monitored are fused to obtain a three-dimensional voxel data set of each plant in the test planting area to be monitored.

2. The method for correcting a digital twin model based on machine vision according to claim 1, characterized in that: Also includes: Extracting features from the three-dimensional voxel data set of each plant in the test planting area to be monitored, respectively, to obtain a plant feature set of each plant in the test planting area to be monitored; Based on the three-dimensional voxel data set and plant feature set of each plant in the experimental planting area to be monitored, the initial digital twin model of the experimental planting area to be monitored pre-stored in the database is matched and corrected to obtain a comprehensive corrected digital twin model of the experimental planting area to be monitored; Based on the comprehensive modified digital twin model of the experimental planting area to be monitored, the real-time plant health index of each plant in the experimental planting area to be monitored is analyzed separately, and a comprehensive analysis is performed to obtain the regional plant real-time health comprehensive index in the experimental planting area to be monitored; The real-time comprehensive health index of regional plants in the experimental planting area to be monitored is judged and analyzed with the preset range of the real-time comprehensive health index of regional plants, and corresponding measures are taken based on the results of the judgment and analysis.

3. A digital twin model correction system based on machine vision, applying the digital twin model correction method based on machine vision according to any one of claims 1-2, characterized in that: include: Data acquisition module, data fusion module, feature extraction module, matching correction module, comprehensive analysis module, monitoring management module; The data acquisition module is used to acquire multispectral image data of several shooting angles of several plants in the test planting area to be monitored in real time and perform preprocessing; The data fusion module is used to fuse the pre-processed multi-spectral image data of each shooting angle of each plant in the test planting area to be monitored to obtain a three-dimensional voxel data set of each plant in the test planting area to be monitored.

4. The machine vision-based digital twin model correction system according to claim 3, characterized in that: Also includes: The feature extraction module is used to extract features from the three-dimensional voxel data set of each plant in the test planting area to be monitored, so as to obtain a plant feature set of each plant in the test planting area to be monitored; The matching and correction module is used to perform matching and correction processing on the initial digital twin model of the test planting area to be monitored pre-stored in the database based on the three-dimensional voxel data set and the plant feature set of each plant in the test planting area to be monitored, so as to obtain a comprehensive and corrected digital twin model of the test planting area to be monitored; The comprehensive analysis module is used to analyze the real-time plant health index of each plant in the test planting area to be monitored based on the comprehensive modified digital twin model of the test planting area to be monitored, and perform comprehensive analysis to obtain the real-time comprehensive health index of the regional plants in the test planting area to be monitored; The monitoring and management module is used to judge and analyze the real-time comprehensive health index of regional plants in the monitored test planting area and the preset real-time comprehensive health index range of regional plants, and take corresponding measures based on the judgment and analysis results.

Images