Real-time Monitoring System and Method for Sesbania cannabina Growth Status Based on Machine Vision

Abstract

The present invention relates to the field of machine vision technology, and discloses a real-time monitoring system and method for the growth state of Sesbania cannabina based on machine vision. The system includes a Sesbania cannabina distributed growth tree generation module, a Sesbania cannabina time-series image generation module, a multi-modal plant feature extraction module, a global growth state analysis module, and a real-time growth state monitoring module. It double-divides the growth area of Sesbania cannabina to obtain a distributed growth tree, and extracts the area corresponding to the leaf nodes; collects multi-modal images at angles and generates time-series images; reconstructs the plant image according to the time-series images, and extracts the multi-modal features of the reconstructed plant image; analyzes the local growth state according to the multi-modal features, and determines the global state according to the local growth state; dynamically generates a monitoring strategy through the local growth state and the global growth state to achieve real-time dynamic monitoring of the growth area of Sesbania cannabina. The present invention can improve the accuracy of monitoring the growth state of plants.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and particularly to a real-time monitoring system and method for the growth state of Sesbania cannabina based on machine vision. Background Art

[0002] With the rapid development of computer technology, computer technology has been gradually applied to the agricultural field. Among them, machine vision technology, as a combination of contemporary computer technology and image sensing network technology, has begun to be applied to the field of crop growth analysis and can provide decision support for judging the growth status of seedling crops. However, in order to analyze the plant growth state more accurately, it is necessary to comprehensively analyze the plant growth area.

[0003] The existing real-time monitoring technology for plant growth state obtains a single plant image through an image acquisition device such as a camera, and then judges the growth state. In practical applications, it is difficult to accurately analyze the plant growth state by only considering a single plant image, resulting in relatively single monitoring of the plant growth state and failure to comprehensively monitor the plant growth state, thus reducing the accuracy of monitoring the plant growth state. Summary of the Invention

[0004] The present invention provides a real-time monitoring system and method for the growth state of Sesbania cannabina based on machine vision, and its main purpose is to solve the problem of relatively low accuracy in monitoring the plant growth state.

[0005] To achieve the above object, a real-time monitoring system for the growth state of Sesbania cannabina based on machine vision provided by the present invention includes:

[0006] A Sesbania cannabina distributed growth tree generation module, configured to perform double partitioning on the pre-acquired Sesbania cannabina growth area to obtain a Sesbania cannabina distributed growth tree, and extract the node growth area corresponding to each leaf node in the Sesbania cannabina distributed growth tree;

[0007] A Sesbania cannabina time series image generation module, configured to collect multi-modal Sesbania cannabina plant images corresponding to the node growth area at a preset rotation angle, and generate a Sesbania cannabina time series image according to the multi-modal Sesbania cannabina plant images;

[0008] A multi-modal plant feature extraction module, configured to reconstruct the Sesbania cannabina plants in the node growth area according to the Sesbania cannabina time series image to obtain a plant reconstruction image, and extract multi-modal plant features corresponding to the plant reconstruction image by using a preset machine vision algorithm;

[0009] A global growth state analysis module, configured to analyze the local growth state of the Sesbania cannabina plants in the node growth area according to the multi-modal plant features, and determine the global growth state of the Sesbania cannabina growth area according to the local growth state;

[0010] A real-time growth status monitoring module, which is used to dynamically generate a Sesbania cannabina growth monitoring strategy based on the local growth status and the global growth status, and use the Sesbania cannabina growth monitoring strategy to perform real-time dynamic monitoring on the Sesbania cannabina growth area to obtain the real-time growth status of Sesbania cannabina plants.

[0011] Optionally, when the Sesbania cannabina distributed growth tree generation module performs double partitioning on the pre-acquired Sesbania cannabina growth area to obtain a Sesbania cannabina distributed growth tree, it is used for:

[0012] Identify the total area of the first area of the pre-acquired Sesbania cannabina growth area, and extract the first highest-order numerical value corresponding to the area value of the total area of the first area;

[0013] Perform a first-level partitioning on the Sesbania cannabina growth area according to the first highest-order numerical value to obtain a first-level growth area;

[0014] Identify the total area of the second area of the first-level growth area, and extract the second highest-order numerical value corresponding to the area value of the total area of the second area;

[0015] Perform a second-level partitioning on the first-level growth area according to the second highest-order numerical value to obtain a second-level growth area;

[0016] Use the Sesbania cannabina growth area, the first-level growth area, and the second-level growth area as area nodes, and construct area paths between the Sesbania cannabina growth area, the first-level growth area, and the second-level growth area;

[0017] Generate a Sesbania cannabina distributed growth tree according to the area nodes and the area paths.

[0018] Optionally, when the Sesbania cannabina time-series image generation module acquires multi-modal Sesbania cannabina plant images corresponding to the node growth area according to a preset rotation angle, it is used for:

[0019] Determine the initial acquisition angle and angle step size of the Sesbania cannabina plants in the node growth area according to the rotation angle;

[0020] Determine acquisition points according to the initial acquisition angle and the angle step size;

[0021] Acquire visible light images, infrared images, and depth images corresponding to the Sesbania cannabina plants in the node growth area through the acquisition points;

[0022] Fuse the visible light image, the infrared image, and the depth image into a multi-modal Sesbania cannabina plant image corresponding to the node growth area, where the multi-modal Sesbania cannabina plant image is:

[0023] ;

[0024] Among them, at the th acquisition point, the pixel points at the moment form a multi-modal Sesbania cannabina plant image, at the th acquisition point, the pixel points at the moment form a visible light image, at the th acquisition point, the pixel points at the moment form an infrared image; at the th acquisition point, the pixel points at the moment form a depth image.

[0025] Optionally, when generating the Sesbania cannabina time series image according to the multi-modal Sesbania cannabina plant image, the Sesbania cannabina time series image generation module is used to:

[0026] Extract the moment points corresponding to the multi-modal Sesbania cannabina plant image;

[0027] Arrange the multi-modal Sesbania cannabina plant images corresponding to the moment points in ascending order of time;

[0028] Use the arranged multi-modal Sesbania cannabina plant images as the Sesbania cannabina time series image.

[0029] Optionally, when reconstructing the Sesbania cannabina plant in the node growth area according to the Sesbania cannabina time series image to obtain a plant reconstruction image, the multi-modal plant feature extraction module is used to:

[0030] Extract the multi-modal Sesbania cannabina plant images at different angles at the same moment in the Sesbania cannabina time series image, and extract the visible light images at different angles in the multi-modal Sesbania cannabina plant images;

[0031] Extract the feature points of the visible light images at different angles one by one, and construct the image correlation of the visible light images at different angles according to the feature points;

[0032] Identify and match the feature points according to the image correlation, and calculate the three-dimensional coordinates of the Sesbania cannabina plant according to the point coordinates of the matched feature points, where the three-dimensional coordinate calculation formula is:

[0033] ;

[0034] Among them, is the three-dimensional coordinate, is the first scale factor, is the second scale factor, is the camera intrinsic matrix, is the visible light image at different angles, represents the zero matrix, is the rotation matrix, is the translation vector, is the point coordinates of the th point coordinates of the

[0035] th matching feature point; Generate the point cloud data of the Sesbania cannabina plant according to the three-dimensional coordinates, generate the triangular mesh of the Sesbania cannabina plant according to the point cloud data, and map the texture information of the visible light image into the triangular mesh to obtain the visible light reconstruction image corresponding to the Sesbania cannabina plant;

[0036] Reconstruct the infrared image and the depth image in the multi-modal Sesbania cannabina plant image;

[0037] Determine the plant reconstruction image according to the reconstructed infrared image, the reconstructed depth image and the visible light reconstruction image.

[0038] Optionally, when the multi-modal plant feature extraction module extracts the multi-modal plant features corresponding to the plant reconstruction image by using a preset machine vision algorithm, it is used for:

[0039] Extract the morphological features of the plant reconstruction image by using a machine vision algorithm, where the morphological features include plant height features, leaf area features and stem diameter features:

[0040] Extract the color features and texture features of the plant reconstruction image;

[0041] Determine the morphological features, the color features and the texture features as the multi-modal plant features.

[0042] Optionally, when the global growth state analysis module analyzes the local growth state of the Sesbania cannabina plant in the node growth area according to the multi-modal plant features, it is used for:

[0043] Compare the morphological features in the multi-modal plant features with the pre-acquired target morphological features to obtain a first comparison factor, and convert the first comparison factor into a first state value;

[0044] Compare the color features in the multi-modal plant features with the pre-acquired target color features to obtain a second comparison factor, and convert the second comparison factor into a second state value;

[0045] Compare the texture feature in the multi-modal plant characteristics with the pre-acquired target texture feature to obtain a third comparison factor, and convert the third comparison factor into a third state value;

[0046] Calculate the state mean values corresponding to the first state value, the second state value, and the third state value, and determine the local growth state of the Sesbania cannabina plants in the node growth area according to the state mean values.

[0047] Optionally, when the real-time growth state monitoring module dynamically generates a Sesbania cannabina growth monitoring strategy through the local growth state and the global growth state, it is used for:

[0048] Determine the first monitoring frequency and monitoring time of the first growth area according to the global growth state;

[0049] Determine the monitoring area of the second growth area through the local growth state, and configure the second monitoring frequency and monitoring depth of the monitoring area;

[0050] Generate a Sesbania cannabina growth monitoring strategy for the first growth area according to the first monitoring frequency and the monitoring time;

[0051] Generate a Sesbania cannabina growth monitoring strategy for the second growth area according to the second monitoring frequency and the monitoring depth.

[0052] Optionally, when the real-time growth state monitoring module uses the Sesbania cannabina growth monitoring strategy to perform real-time dynamic monitoring on the Sesbania cannabina growth area to obtain the real-time growth state of the Sesbania cannabina plants, it is used for:

[0053] Use the Sesbania cannabina growth monitoring strategy to collect real-time growth image data of the Sesbania cannabina growth area in real time;

[0054] Analyze the state quantization values corresponding to the real-time growth image data;

[0055] Determine the real-time growth state of the Sesbania cannabina plants according to the analyzed state quantization values.

[0056] To solve the above problems, the present invention also provides a method for real-time monitoring of the growth state of Sesbania cannabina based on machine vision, and the method includes:

[0057] Perform double partitioning on the pre-acquired Sesbania cannabina growth area to obtain a Sesbania cannabina distributed growth tree, and extract the node growth areas corresponding to each leaf node in the Sesbania cannabina distributed growth tree;

[0058] Collect multi-modal Sesbania cannabina plant images corresponding to the node growth area at a preset rotation angle, and generate a Sesbania cannabina time-series image according to the multi-modal Sesbania cannabina plant images;

[0059] Reconstruct the Sesbania plants in the node growth area according to the Sesbania time series images to obtain plant reconstruction images, and use a preset machine vision algorithm to extract the multi-modal plant features corresponding to the plant reconstruction images;

[0060] Analyze the local growth state of the Sesbania plants in the node growth area according to the multi-modal plant features, and determine the global growth state of the Sesbania growth area according to the local growth state;

[0061] Dynamically generate a Sesbania growth monitoring strategy through the local growth state and the global growth state, and use the Sesbania growth monitoring strategy to perform real-time dynamic monitoring on the Sesbania growth area to obtain the real-time growth state of the Sesbania plants.

[0062] In the embodiment of the present invention, the Sesbania growth area is divided into two parts, a Sesbania distributed growth tree is constructed, and then refined monitoring is realized, and targeted analysis is carried out according to the characteristics of different local areas; multi-modal Sesbania plant images are collected according to the rotation angle, covering multi-dimensional information of Sesbania at different angles, overcoming the limitations of a single perspective and data type; the Sesbania plants are reconstructed according to the time series images to obtain plant reconstruction images that more comprehensively and accurately reflect the true form of Sesbania; based on the multi-modal plant features, the local growth state of the Sesbania plants in the node growth area is analyzed, and the health status, growth rate, etc. of Sesbania in each small area can be analyzed, and then the global growth state is determined according to the local growth state, taking into account both the overall trend and local differences; the Sesbania growth monitoring strategy is dynamically generated through the local and global growth states, so that the monitoring strategy can be adjusted in real time according to the actual growth situation of Sesbania, ensuring the timeliness and accuracy of monitoring. Therefore, the real-time monitoring system and method for Sesbania growth state based on machine vision proposed by the present invention can solve the problem of low accuracy in monitoring the growth state of plants. Description of the Drawings

[0063] Figure 1 It is a functional module diagram of a real-time monitoring system for Sesbania growth state based on machine vision provided by an embodiment of the present invention;

[0064] Figure 2 It is a schematic flowchart of an operation method of a real-time monitoring system for Sesbania growth state based on machine vision provided by an embodiment of the present invention.

[0065] The realization, functional characteristics and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the drawings. Detailed Embodiments

[0066] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0067] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0068] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. "Plural" generally includes at least two.

[0069] Depending on the context, the words "if", "when" as used herein may be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" may be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0070] In addition, the step timings in the following method embodiments are only examples and are not strictly limited.

[0071] In fact, the server devices deployed in the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision may be composed of one or more devices. The above-mentioned real-time monitoring system for the growth state of Sesbania cannabina based on machine vision can be implemented as: a service instance, a virtual machine, or a hardware device. For example, the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision can be implemented as a service instance deployed on one or more devices in a cloud node. Briefly, the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision can be understood as a software deployed on a cloud node for providing the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision to each client. Alternatively, the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision can also be implemented as a virtual machine deployed on one or more devices in a cloud node. An application software for managing each client is installed in the virtual machine. Alternatively, the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision can also be implemented as a server composed of many identical or different types of hardware devices, and one or more hardware devices are set to provide the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision to each client.

[0072] In terms of implementation form, the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision adapts to the user side. That is, if the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision is an application installed on the cloud service platform, then the user side is the client that establishes a communication connection with this application; or if the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision is implemented as a website, then the user side is implemented as a web page; or if the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision is implemented as a cloud service platform, then the user side is implemented as a small program in an instant messaging application.

[0073] Refer to Figure 1 As shown, it is a functional module diagram of the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision provided by an embodiment of the present invention.

[0074] The real-time monitoring system 100 for the growth status of Sesbania cannabina based on machine vision according to the present invention can be set in a cloud server. In terms of implementation form, it can be one or more service devices, or can be an application installed on the cloud (such as the server of a mobile service operator, a server cluster, etc.), or can also be developed into a website. According to the functions to be realized, the real-time monitoring system 100 for the growth status of Sesbania cannabina based on machine vision can include a Sesbania cannabina distributed growth tree generation module 101, a Sesbania cannabina time series image generation module 102, a multi-modal plant feature extraction module 103, a global growth status analysis module 104, and a real-time growth status monitoring module 105. The modules in the present invention can also be called units, which refer to a series of computer program segments that can be executed by a device processor and can complete fixed functions, and are stored in the memory of the device.

[0075] In the embodiment of the present invention, in the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision, each of the above modules can be independently implemented and called with other modules. Here, the call can be understood as that a certain module can be connected to multiple modules of another type and provide corresponding services for the multiple modules it is connected to. For example, the sharing and evaluation module can call the same information collection module to obtain the information collected by this information collection module. Based on the above characteristics, in the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision provided by the embodiment of the present invention, without modifying the program code, the applicable range of the architecture of the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision can be adjusted by adding modules and directly calling, so as to achieve cluster-level horizontal expansion, so as to achieve the purpose of quickly and flexibly expanding the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision. In practical applications, the above modules can be set in the same device or different devices, or can also be set in virtual devices, such as service instances in a cloud server.

[0076] The following describes each component and the specific working process of the real-time monitoring system for the growth status of Sesbania cannabina based on machine vision in combination with specific embodiments:

[0077] The Sesbania cannabina distributed growth tree generation module 101 is used to perform a two-fold division on the pre-acquired Sesbania cannabina growth area to obtain a Sesbania cannabina distributed growth tree, and extract the node growth area corresponding to each leaf node in the Sesbania cannabina distributed growth tree.

[0078] In the embodiment of the present invention, the Sesbania cannabina distributed growth tree is a tree structure, with the Sesbania cannabina growth area as the root node, the first-level growth area as the first-level child nodes, and the second-level growth area as the second-level child nodes. Each node is connected by a region path, clearly showing the hierarchical division structure of the Sesbania cannabina growth area, which helps to systematically manage and monitor the Sesbania cannabina growth areas at different levels, and is convenient for quickly locating and analyzing the growth status of Sesbania cannabina in a specific area.

[0079] In the embodiment of the present invention, when the Sesbania cannabina distributed growth tree generation module 101 performs a two-fold division on the pre-acquired Sesbania cannabina growth area to obtain a Sesbania cannabina distributed growth tree, it is used for:

[0080] Identify the total area of the first area of the pre-acquired Sesbania cannabina growth area, and extract the first highest-order numerical value corresponding to the area value of the total area of the first area;

[0081] Perform a first-fold division on the Sesbania cannabina growth area according to the first highest-order numerical value to obtain a first-level growth area;

[0082] Identify the total area of the second area of the first-level growth area, and extract the second highest-order numerical value corresponding to the area value of the total area of the second area;

[0083] Perform a second-fold division on the first-level growth area according to the second highest-order numerical value to obtain a second-level growth area;

[0084] Take the Sesbania cannabina growth area, the first-level growth area, and the second-level growth area as region nodes, and construct a region path between the Sesbania cannabina growth area, the first-level growth area, and the second-level growth area;

[0085] Generate a Sesbania cannabina distributed growth tree according to the region nodes and the region path.

[0086] Specifically, identify the total area of the pre-acquired Sesbania cannabina growth area, that is, the total area of the first region. Then, satellite remote sensing images can be used to obtain images of the Sesbania cannabina growth area. Through image processing software, combined with GIS technology, according to the coordinate information and scale on the image, the area of the region can be accurately calculated. For example, if the total area of the first region is 5000, extract the highest digit value corresponding to the area value of the total area of the first region, that is, the first highest digit value. Since the highest digit value of 5000 is 5, the highest digit value will be used as the basis for the number of portions in the first-level division. Divide the Sesbania cannabina growth area into equal-area portions according to the first highest digit value of 5, obtaining 5 first-level growth regions. That is, the large-scale Sesbania cannabina growth area is initially refined, facilitating more accurate monitoring of different local regions in the follow-up and helping to discover differences in Sesbania cannabina growth at a larger scale.

[0087] Specifically, for each region obtained from the first-level division, identify its total area of the second region respectively. Assume that the area of one of the first-level growth regions is 400, and extract the second highest digit value of 4 corresponding to its area value. This value will be used for the second-level division of the corresponding first-level growth region. Divide the corresponding first-level growth region into 4 portions according to the second highest digit value of 4, obtaining the second-level growth regions. Then, the Sesbania cannabina growth area is further subdivided, enabling monitoring to penetrate into smaller local ranges and capturing subtle changes in Sesbania cannabina growth.

[0088] Exemplarily, the Sesbania cannabina growth area is , the first-level growth region is , the second-level growth region is , etc. Furthermore, take the Sesbania cannabina growth area as the root node, the first-level growth region as the first-level child nodes, and the second-level growth regions , as the second-level child nodes, and clarify the hierarchical relationship and subordination relationship between them. For example, contains , and in turn contains , thus generating the Sesbania cannabina distributed growth tree.

[0089] Furthermore, extract the node growth regions corresponding to each leaf node in the Sesbania cannabina distributed growth tree. Then, starting from the root node of the Sesbania cannabina distributed growth tree, search downward along the branches of the tree until reaching the leaf node. When traversing to the leaf node, determine the actual growth region corresponding to the leaf node according to the recorded node information. Since the corresponding relationship between each regional node and the actual Sesbania cannabina growth region has been clarified when constructing the Sesbania cannabina distributed growth tree, the corresponding node growth region can be found through information such as the name or number of the leaf node. For example, the leaf node When constructing the tree, it is obtained from the first growth area through the second division. Then, according to the previous division rules and records, the position and range of the specific Sesbania cannabina growth area represented can be determined.

[0090] Furthermore, the Sesbania cannabina plant has a three-dimensional structure. Capturing images from a single angle can only obtain partial information, resulting in information loss. Therefore, the Sesbania cannabina plant needs to be photographed from multiple perspectives to avoid missing important features due to a single perspective.

[0091] The Sesbania cannabina time series image generation module 102 is used to collect multi-modal Sesbania cannabina plant images corresponding to the node growth area according to a preset rotation angle, and generate Sesbania cannabina time series images based on the multi-modal Sesbania cannabina plant images.

[0092] In the embodiment of the present invention, the multi-modal Sesbania cannabina plant image refers to an image obtained by fusing image information of the Sesbania cannabina plant from different sensors (such as visible light cameras, infrared cameras, depth cameras), which contains information of the Sesbania cannabina plant in multiple modalities of visible light, infrared light, and depth, and can describe the characteristics and states of the Sesbania cannabina plant more comprehensively and richly.

[0093] In the embodiment of the present invention, when the Sesbania cannabina time series image generation module 102 collects multi-modal Sesbania cannabina plant images corresponding to the node growth area according to a preset rotation angle, it is used for:

[0094] Determining the initial acquisition angle and angle step size of the Sesbania cannabina plant in the node growth area according to the rotation angle;

[0095] Determining acquisition points according to the initial acquisition angle and the angle step size;

[0096] Collecting visible light images, infrared images, and depth images of the Sesbania cannabina plant corresponding to the node growth area through the acquisition points;

[0097] Fusing the visible light image, the infrared image, and the depth image into a multi-modal Sesbania cannabina plant image corresponding to the node growth area, where the multi-modal Sesbania cannabina plant image is:

[0098] ;

[0099] where, is the pixel point at the th acquisition point at the th moment that composes the multi-modal Sesbania cannabina plant image, is the pixel point at the th moment at the The visible light image formed is the pixel point at the th acquisition point at the th moment that forms the infrared image; is the pixel point at the th acquisition point at the th moment that forms the depth image.

[0100] Specifically, when collecting images of Sesbania cannabina plants in the node growth area, the initial acquisition angle is determined based on the rotation angle, that is, the angle when starting to collect images. If we want to comprehensively collect images of Sesbania cannabina plants from different angles, the initial acquisition angle may be set to 0 degrees, and the angle step is the angle interval of rotation after each image collection. For example, images are collected every 30 degrees of rotation, so as to ensure obtaining image information of Sesbania cannabina plants from multiple angles. Then, the initial acquisition angle determines the starting direction of collection, and the angle step stipulates the interval of each rotation. Starting from the initial acquisition angle and successively rotating by the angles stipulated by the angle step, a series of acquisition points can be determined. For example, if the initial acquisition angle is 0 degrees and the angle step is 30 degrees, then the acquisition points are 0 degrees, 30 degrees, 60 degrees, 90 degrees, etc. The acquisition points represent the positions of collecting images of Sesbania cannabina plants at different angles.

[0101] Specifically, after determining the acquisition points, at the position of each acquisition point, corresponding devices (such as visible light cameras, infrared cameras, depth cameras, etc.) are used to collect images of Sesbania cannabina plants in the node growth area. Then, the visible light image can reflect the appearance information of Sesbania cannabina plants, such as color, shape, texture, etc.; the infrared image mainly reflects the temperature distribution on the plant surface, which helps to understand the physiological state of the plant; the depth image provides three-dimensional space information about the plant, such as the distance and depth of each part of the plant. Thus, for the images collected at the th moment at each acquisition point, the visible light image, infrared image, and depth image are fused. The multi-modal Sesbania cannabina plant image is a set that contains image information of three different modalities. Therefore, using image information of different modalities, more in-depth information about Sesbania cannabina plants that cannot be provided by a single-modal image can be obtained.

[0102] Furthermore, the growth of Sesbania cannabina is a dynamic process that changes over time. The multi-modal Sesbania cannabina plant image can provide rich information at a certain moment, but a single image cannot intuitively reflect the continuity and change trend of growth. Therefore, it is necessary to analyze the continuous time period of Sesbania cannabina growth.

[0103] In the embodiments of the present invention, the Sesbania time-series image is a multi-modal Sesbania plant image arranged in chronological order, representing the corresponding multi-modal plant images at different moments, recording the state information of the Sesbania plant at different time points. Through the Sesbania time-series image, the changes in the morphology, color, texture, etc. of the Sesbania plant over time, as well as the feature changes under different modalities (visible light, infrared, depth, etc.), can be intuitively seen.

[0104] In the embodiments of the present invention, when generating the Sesbania time-series image according to the multi-modal Sesbania plant image, the Sesbania time-series image generation module 102 is used for:

[0105] Extract the time points corresponding to the multi-modal Sesbania plant images;

[0106] Arrange the multi-modal Sesbania plant images corresponding to the time points in ascending order of time;

[0107] Use the arranged multi-modal Sesbania plant images as the Sesbania time-series image.

[0108] Specifically, the multi-modal Sesbania plant images are collected at different times, and each image corresponds to a specific collection time point. After extracting the time points of all multi-modal Sesbania plant images, these time points are compared and sorted. Starting from the earliest time point in chronological order, the corresponding multi-modal Sesbania plant images are arranged in sequence to ensure the time continuity between the images, so that the subsequent changes of the Sesbania plant in the time dimension can be clearly observed. Thus, after arranging the multi-modal Sesbania plant images in ascending order of time, they form the Sesbania time-series image.

[0109] Furthermore, the Sesbania time-series image contains the multi-modal Sesbania plant image information at different time points and multiple angles. In order to more realistically reflect the actual morphology and structure of the plant, it is necessary to reconstruct the three-dimensional model of the Sesbania plant.

[0110] The multi-modal plant feature extraction module 103 is used to reconstruct the Sesbania plant in the node growth area according to the Sesbania time-series image to obtain a plant reconstruction image, and use a preset machine vision algorithm to extract the multi-modal plant features corresponding to the plant reconstruction image.

[0111] In the embodiments of the present invention, the plant reconstruction image refers to a three-dimensional image generated by processing and analyzing the multi-modal images (visible light, infrared, depth, etc.) in the Sesbania time-series image, which can intuitively display the morphology, structure and physiological state of the Sesbania plant.

[0112] In the embodiment of the present invention, when the multimodal plant feature extraction module 103 reconstructs the sesbania plants in the node growth region according to the sesbania time series images to obtain a plant reconstruction image, it is used for:

[0113] Extract multimodal sesbania plant images at different angles at the same moment in the sesbania time series images, and extract visible light images at different angles in the multimodal sesbania plant images;

[0114] Extract the feature points of the visible light images at different angles one by one, and construct the image association of the visible light images at different angles according to the feature points;

[0115] Identify the matching feature points according to the image association, and calculate the three-dimensional coordinates of the sesbania plants according to the point coordinates of the matching feature points. The three-dimensional coordinate calculation formula is:

[0116] ;

[0117] Wherein, is the three-dimensional coordinate, is the first scaling factor, is the second scaling factor, is the camera internal parameter matrix, is the visible light image at different angles, represents the zero matrix, is the rotation matrix, is the translation vector, is the point coordinate of the th matching feature point, is the point coordinate of the

[0118] th matching feature point;

[0118] Generate the point cloud data of the sesbania plants according to the three-dimensional coordinates, generate the triangular mesh of the sesbania plants according to the point cloud data, and map the texture information of the visible light image to the triangular mesh to obtain the visible light reconstruction image corresponding to the sesbania plants;

[0119] Reconstruct the infrared image and depth image in the multimodal sesbania plant image;

[0120] Determine the plant reconstruction image according to the reconstructed infrared image, the reconstructed depth image and the visible light reconstruction image.

[0121] Specifically, from the sesbania time series images, identify multimodal sesbania plant images at different angles at the same moment. That is, the images at the same moment can reflect the state of the plants at that moment, and the images at different angles provide multi-perspective information of the plants, so as to extract the visible light images at different angles at the same moment. For example, at moment, angle The corresponding visible light image is , at an angle The corresponding visible light image is , at an angle The corresponding visible light image is and so on, until all the visible light images at different angles are extracted to obtain the visible light images at different angles, and for the multi-modal Sesbania cannabina plant images at different collection points and times , the Scale-Invariant Feature Transform (SIFT) algorithm is used to extract feature points, and the feature points are obtained by constructing scale-space extrema detection. For an image , its scale space is defined as , where is the Gaussian kernel function is the scale factor represents the convolution operation, and the Difference of Gaussian (DoG) function at different scales is used to detect feature points, then is a constant

[0122] Specifically, the Euclidean distance is used to match the feature points between the visible light images at different angles. For two feature points and , their Euclidean distance is , then the pair of feature points with the closest distance is identified, the connection between the images from different perspectives and times is established, and triangulation is performed based on the feature points. Assuming the camera intrinsic matrix is , for the matched pair of feature points, in the visible light image and the visible light image , let the corresponding image point coordinates be and , respectively. Then the three-dimensional point coordinates satisfy , , where and are scale factors is the rotation matrix is the translation vector represents a two-dimensional vector used for a specific transformation that projects the coordinates of a three-dimensional space point into a two-dimensional space. Among them, in represents the visible light image matrix at different angles represents the zero matrix, and the three-dimensional point coordinates are obtained by solving the system of equations. Furthermore, the three-dimensional coordinates obtained through triangulation are combined into a point cloud, and the Delaunay triangulation algorithm is used to generate a triangular mesh on the optimized point cloud. For the point cloud set ​, Delaunay triangulation ensures that the circumcircle of each triangle does not contain other points, and then maps the texture information of the multi-modal Sesbania cannabina plant image onto the mesh model. According to the corresponding relationship of the feature points, the color, texture and other information of the images at different acquisition points and times are accurately fitted onto the surface of the 3D model, and finally a 3D reconstruction image of the Sesbania cannabina plant is obtained.

[0123] Furthermore, reconstruct the infrared image and depth image in the multi-modal Sesbania cannabina plant image. The reconstruction steps are the same as those of the visible light image. Among them, the infrared image reconstruction may pay more attention to the accurate restoration of temperature information, and the depth image reconstruction focuses on the accurate representation of three-dimensional space distance information. Then, determine the plant reconstruction image according to the reconstructed infrared image, reconstructed depth image and visible light reconstruction image. The plant reconstruction image is a 3D image that synthesizes multi-modal information such as visible light, infrared and depth, and it contains multi-faceted information such as the appearance, temperature distribution and three-dimensional space structure of the Sesbania cannabina plant.

[0124] Even further, the plant reconstruction image contains multi-modal information such as visible light, infrared, and depth, can reflect the characteristics of the Sesbania cannabina plant from different angles, synthesize multi-faceted information, and comprehensively evaluate the growth state of the Sesbania cannabina plant.

[0125] In the embodiments of the present invention, the multi-modal plant characteristics include the morphological characteristics, color characteristics and texture characteristics of the Sesbania cannabina plant. The morphological characteristics describe the external morphology of the Sesbania cannabina through geometric parameters (plant height, leaf area, stem diameter); the color characteristics reflect the health status of the leaves through the color mean value; the texture characteristics describe the microscopic structure of the leaves through the gray-level co-occurrence matrix.

[0126] In the embodiments of the present invention, when the multi-modal plant feature extraction module 103 extracts the multi-modal plant features corresponding to the plant reconstruction image by using a preset machine vision algorithm, it is used for:

[0127] Extract the morphological characteristics of the plant reconstruction image by using the following machine vision algorithm:

[0128] ;

[0129] ;

[0130] ;

[0131] Among them, is the plant height feature in the morphological characteristics at the th moment, is the leaf area feature in the morphological characteristics at the th moment, is the stem diameter feature in the morphological characteristics at the The ordinate of the highest point of the Sesbania cannabina plant in the reconstructed image of the plant is the plant height control factor is the leaf area control factor is the stem diameter control factor in the reconstructed image of the plant is the pixel value at the position The abscissa of the Sesbania cannabina plant in the reconstructed image of the plant is the width of the reconstructed image of the plant is the length of the reconstructed image of the plant

[0132] Extract the color features and texture features of the reconstructed image of the plant

[0133] Determine the morphological features, the color features, and the texture features as multimodal plant features

[0134] Specifically, the morphological features include plant height, leaf area, and stem diameter. Then the plant height calculation formula is , is the plant height of Sesbania cannabina at time , is the ordinate of the highest point of Sesbania cannabina in the image is the ordinate of the lowest point of Sesbania cannabina in the image, that is, calculate the height of Sesbania cannabina through vertical projection to reflect its growth state. The leaf area calculation formula is , is the leaf area of Sesbania cannabina at time , is the pixel value at the position in the binary image (1 represents Sesbania cannabina, 0 represents the background), represents the sum of all pixels in the image, that is, estimate the leaf area by counting the number of pixels of Sesbania cannabina in the binary image to reflect the growth of the leaves; the stem diameter calculation formula is , is the stem diameter of Sesbania cannabina at time , is the abscissa of the rightmost point of Sesbania cannabina in the image is the abscissa of the leftmost point of Sesbania cannabina in the image, that is, calculate the stem diameter of Sesbania cannabina through horizontal projection to reflect the development of the stem. And for the control factors in the machine vision algorithm , and determine the contribution degrees of the plant height feature, the leaf area feature, and the stem diameter feature to the morphological features. If it is desired that the plant height feature dominates in the comprehensive morphological features, the value of can be appropriately increased; if it is desired to highlight the influence of the leaf area feature, the value of The value, by adjusting the control factor, flexibly adjusts the importance of each morphological feature in the comprehensive evaluation. That is, when calculating the plant height feature in the morphological features, the plant height control factor is set to 1, and the leaf area control factor and the stem diameter control factor are set to 0; while when calculating the leaf area feature in the morphological features, the leaf area control factor is set to 1, and the plant height control factor and the stem diameter control factor are set to 0; when calculating the stem diameter feature in the morphological features, the stem diameter control factor is set to 1, and the plant height control factor and the leaf area control factor are set to 0, thus realizing the independent calculation and analysis of the three morphological features of plant height, leaf area and stem diameter.

[0135] Specifically, the color feature refers to the average value of the leaf color, then the average value of the leaf color is calculated as , is the average value of the leaf color of Sesbania cannabina at time , is the color value (value in the RGB or HSV color space) at the position in the image, is the total number of pixels of Sesbania cannabina in the image, is the sum of all pixels in the image, that is, by calculating the average value of the leaf color, it reflects the health status of the leaf (such as whether it is yellowing or withering), and the texture feature is obtained through the gray-level co-occurrence matrix, that is , is the value at the position in the gray-level co-occurrence matrix, indicating the frequency of co-occurrence of the gray level and in a specific direction, is the gray value at the position in the image, is an indicator function, which is 1 when the condition in the parentheses holds, otherwise 0, that is , that is, by statistically analyzing the frequency of gray-level co-occurrence, the texture features of Sesbania cannabina leaves are extracted, reflecting the roughness, uniformity, etc. of the leaves, and thus the morphological features, color features and texture features are determined as multi-modal plant features.

[0136] Furthermore, the multi-modal plant features cover various aspects of information such as the morphology, color, texture, temperature, and spatial structure of the plant. By integrating the multi-modal features, the true growth state of Sesbania cannabina plants in the node growth area can be comprehensively and meticulously reflected, avoiding the limitations of single-feature analysis.

[0137] The global growth state analysis module 104 is configured to analyze the local growth state of the Sesbania cannabina plants in the node growth region according to the multi-modal plant characteristics, and determine the global growth state of the Sesbania cannabina growth region according to the local growth state.

[0138] In an embodiment of the present invention, the local growth state refers to the growth state of the Sesbania cannabina plants in the node growth region corresponding to the leaf nodes.

[0139] In an embodiment of the present invention, when analyzing the local growth state of the Sesbania cannabina plants in the node growth region according to the multi-modal plant characteristics, the global growth state analysis module 104 is configured to:

[0140] Compare the morphological characteristics in the multi-modal plant characteristics with the pre-acquired target morphological characteristics to obtain a first comparison factor, and convert the first comparison factor into a first state value;

[0141] Compare the color characteristics in the multi-modal plant characteristics with the pre-acquired target color characteristics to obtain a second comparison factor, and convert the second comparison factor into a second state value;

[0142] Compare the texture characteristics in the multi-modal plant characteristics with the pre-acquired target texture characteristics to obtain a third comparison factor, and convert the third comparison factor into a third state value;

[0143] Calculate the state mean corresponding to the first state value, the second state value, and the third state value, and determine the local growth state of the Sesbania cannabina plants in the node growth region according to the state mean.

[0144] Specifically, the morphological characteristics in the multi-modal plant characteristics include the plant height, stem diameter, and leaf area of the Sesbania cannabina plant. The pre-acquired target morphological characteristics are the morphological standards that a Sesbania cannabina plant should have in the growth area at this node under an ideal state. Then, the actually extracted morphological characteristics are compared with the target morphological characteristics to calculate the degree of difference between the two, obtaining the first comparison factor. For example, if the target plant height is 50 cm and the actually measured plant height is 40 cm, there is a difference in the plant height index. The first comparison factor is represented by the difference between the two. Then, the first comparison factor is converted into the first state value. If the difference value of the first comparison factor is less than the preset threshold, the first state value is marked as 1; otherwise, it is marked as 0. That is, if the first comparison factor is small, it indicates that the actual morphological characteristics are relatively close to the target morphological characteristics, and the converted first state value is relatively high; conversely, if the difference is large, the first state value is low. The color characteristics reflect the color information of the leaves, stems, and other parts of the Sesbania cannabina plant, such as the green degree of the leaves, whether there is yellowing or color change, etc. The target color characteristics are also the preset ideal color standards. The actual color characteristics are compared with the target color characteristics, and the second comparison factor is obtained by analyzing the difference in the leaf color mean parameter. Then, the second comparison factor is converted into the second state value according to the threshold conversion rule to represent the compliance degree of the color characteristics within a unified numerical range. The threshold conversion rule is that when the difference value of the comparison factor is less than the preset threshold, the state value is marked as 1; otherwise, it is marked as 0.

[0145] Specifically, the texture characteristics reflect the texture details on the surface of the Sesbania cannabina plant, such as the clarity of the leaf veins, whether there are damages or disease spots, etc. The target texture characteristics are the texture standards under the ideal state. The actual texture characteristics are compared with the target texture characteristics to obtain the third comparison factor. The third comparison factor is converted into the third state value according to the threshold conversion rule to measure the state of the texture characteristics, and the state mean corresponding to the first state value, the second state value, and the third state value is calculated. According to this state mean, the local growth state of the Sesbania cannabina plant in the node growth area is determined. For example, if the state mean is the value 1, it indicates that the local growth state is a healthy state; if the state mean is 2 / 3, the local growth state is a sub-healthy state; if the state mean is 1 / 3, the local growth state is a poor state; if the state mean is 0, the local growth state is a severely poor state.

[0146] Furthermore, the Sesbania cannabina growth area consists of multiple node growth areas. The local growth state of each node growth area is a specific manifestation of the growth situation of Sesbania cannabina in this specific area. By integrating the local information, the situation of the entire growth area can be understood.

[0147] In an embodiment of the present invention, the global growth state refers to the growth state of Sesbania cannabina in the growth area corresponding to the root node in the Sesbania cannabina distributed growth tree determined according to the local growth states of the leaf nodes in the Sesbania cannabina distributed growth tree.

[0148] In an embodiment of the present invention, when determining the global growth state of the Sesbania cannabina growth area according to the local growth state, the global growth state analysis module 104 is configured to:

[0149] Extract the state mean value corresponding to the local growth state;

[0150] Calculate the global growth state quantization value of the Sesbania cannabina growth area according to the state mean value:

[0151] ;

[0152] Wherein, is the global growth state quantization value, is the first-level growth area the second-level growth area in the corresponding state mean value, is the first-level growth area the second-level growth area in the corresponding state value correction coefficient, is the total number of features of the multi-modal plant characteristics, is the number of areas of the first-level growth area, is the first-level growth area the number of areas of the corresponding second-level growth area;

[0153] Determine the global growth state of the Sesbania cannabina growth area according to the global growth state quantization value.

[0154] Specifically, the local state mean value corresponding to the overall second-level growth area is calculated according to the local growth states of the Sesbania cannabina plants in each node growth area (i.e., the second-level growth area). For example, each leaf node in the second-level growth area corresponds to a state value, and the mean value of all the state values corresponding to the leaf nodes is calculated to obtain the local state mean value. Thus, the local state mean values corresponding to the nodes in all the first-level growth areas are obtained according to the leaf nodes, and then the global growth state quantization value of the Sesbania cannabina growth area is calculated according to the local state mean value.

[0155] Specifically, is to adjust the local state mean value. If there is an error in judging the state value of the Sesbania cannabina growth area corresponding to the leaf node, for example, the state value 1 is judged as the state value 0, or the state value 0 is judged as the state value 1, then the state value correction coefficient and the total number of features are used for adjustment. For example, the corresponding state value is 1, The corresponding status value is 1 / 3, The corresponding status value is 0, The local status mean value of is 4 / 9, while After verification, the corresponding status value should be 2 / 3. At this time, it means that two features in the multi-modal plant characteristics are in a healthy state, while one is not. Then the status value correction coefficient The value of is 2. Furthermore, according to the calculated global growth status quantization value, the global growth status of the Sesbania cannabina growth area is determined. When the global growth status quantization value is 1, it indicates that the global growth status of the Sesbania cannabina growth area is good. When the global growth status quantization value is 0, it indicates that the global growth status of the Sesbania cannabina growth area is poor. When the global growth status quantization value is less than 1 and greater than 0, it indicates that the global growth status of the Sesbania cannabina growth area is average.

[0156] Exemplarily, in a tree structure, if the root node is A, the first-level nodes are A1 and A2, and the leaf nodes are A11, A12, A13 corresponding to A1 and A21, A22 corresponding to A2. Each leaf node corresponds to a value. The average value of the values of the leaf nodes corresponding to A1 is taken as the value of A1, the average value of the values of the leaf nodes corresponding to A2 is taken as the value of A2, and then the average value of the values of A1 and A2 is taken as the value of the root node A. Then the root node A represents the global growth status of the Sesbania cannabina growth area.

[0157] Furthermore, the growth of Sesbania cannabina is a dynamic process, and there are differences in the growth conditions at different growth stages and different local areas. Therefore, different local growth statuses and global growth statuses require different monitoring methods to improve the pertinence and effectiveness of the monitoring strategy.

[0158] The real-time growth status monitoring module 105 is used to dynamically generate a Sesbania cannabina growth monitoring strategy based on the local growth status and the global growth status, and use the Sesbania cannabina growth monitoring strategy to perform real-time dynamic monitoring on the Sesbania cannabina growth area to obtain the real-time growth status of the Sesbania cannabina plants.

[0159] In the embodiment of the present invention, the Sesbania cannabina growth monitoring strategy refers to a series of monitoring plans formulated to comprehensively and accurately master the growth situation of Sesbania cannabina. Through a reasonable monitoring strategy, problems occurring in the growth process of Sesbania cannabina can be discovered in time, the healthy growth of Sesbania cannabina can be guaranteed, and the yield and quality can be improved.

[0160] In the embodiment of the present invention, when the real-time growth status monitoring module 105 dynamically generates a Sesbania cannabina growth monitoring strategy through the local growth status and the global growth status, it is used for:

[0161] Determine the first monitoring frequency and monitoring time of the first growth area according to the global growth status;

[0162] Determine the monitoring area of the second growth region based on the local growth state, and configure the second monitoring frequency and monitoring depth of the monitoring area;

[0163] Generate a Sesbania cannabina growth monitoring strategy for the first growth region according to the first monitoring frequency and the monitoring time;

[0164] Generate a Sesbania cannabina growth monitoring strategy for the second growth region according to the second monitoring frequency and the monitoring depth.

[0165] Specifically, the global growth state reflects the comprehensive growth situation of the entire Sesbania cannabina growth region. If the global growth state is good, indicating that the overall growth of Sesbania cannabina is relatively stable, the first monitoring frequency of the first growth region (the larger region obtained by initially dividing the Sesbania cannabina growth region) can be relatively low. For example, if it was originally planned to monitor once a day, it can now be adjusted to once every three days. On the contrary, if the global growth state is not good, such as the overall growth rate of Sesbania cannabina is slow, there are many signs of pests and diseases, etc., it is necessary to increase the monitoring frequency, which may increase from once a day to multiple times a day in order to timely grasp the changes in the growth of Sesbania cannabina and take corresponding measures; the determination of the monitoring time is also related to the global growth state. When the global growth state is good, the monitoring time can be selected during the regular time period, such as monitoring during the day when the light is sufficient, which is convenient for obtaining clear images and data. However, if the global growth state is abnormal, it may be necessary to adjust the monitoring time according to the specific situation. If it is suspected that Sesbania cannabina is affected by the low temperature at night, it is necessary to increase the monitoring during specific time periods at night to determine the impact of temperature changes on the growth of Sesbania cannabina.

[0166] Specifically, the local growth state shows the specific growth conditions of the second-level growth region (a smaller region further subdivided on the basis of the first-level growth region). By analyzing the local growth state, regions with abnormal growth states are identified as key monitoring regions. For example, if the Sesbania cannabina plants in certain regions show phenomena such as yellowing and withering of leaves, these regions are determined as monitoring regions. Thus, for the determined monitoring regions, the second monitoring frequency is configured according to the degree of abnormality of their local growth states. If the growth state in this region is severely abnormal, such as in a region where pests and diseases break out, the monitoring frequency needs to be significantly increased, perhaps once per hour; while for regions with slightly abnormal growth states, the monitoring frequency can be relatively lower, such as twice a day. The monitoring depth refers to the detailed degree of obtaining Sesbania cannabina growth information within the monitoring region. For monitoring regions with severely abnormal growth states, the monitoring depth needs to be increased. For example, not only the appearance of Sesbania cannabina plants needs to be monitored, but also internal factors such as soil nutrients and root growth need to be analyzed in depth. Thus, by combining the previously determined first monitoring frequency and monitoring time, a Sesbania cannabina growth monitoring strategy for the first-level growth region is generated. For example, if the first monitoring frequency is once every three days and the monitoring time is from 9 am to 11 am every day, then the monitoring strategy for this region is to conduct a comprehensive monitoring of the first-level growth region from 9 am to 11 am every three days. The monitoring content includes using drones to collect large-area image data to obtain information such as the overall growth trend of Sesbania cannabina. According to the determined second monitoring frequency and monitoring depth, a Sesbania cannabina growth monitoring strategy for the second-level growth region is generated. For example, if the second monitoring frequency is twice a day, in the morning and afternoon respectively, and the monitoring depth is to comprehensively analyze soil nutrients and plant physiological indicators, then the monitoring strategy for this region is to conduct a detailed monitoring of the determined monitoring region in the morning and afternoon every day, including collecting soil samples for nutrient analysis and using professional instruments to detect physiological indicators such as the photosynthetic rate and transpiration rate of plants.

[0167] Furthermore, by grasping the growth trend of Sesbania cannabina plants in real time according to the Sesbania cannabina growth monitoring strategy, such as whether the growth speed is accelerating or slowing down, it helps to judge whether the current growth environment is suitable and whether Sesbania cannabina is developing towards the expected growth goal.

[0168] In the embodiment of the present invention, the real-time growth state refers to the actual growth conditions of Sesbania cannabina plants at the current moment, including the characteristic manifestations of Sesbania cannabina plants in terms of morphology, color, texture, etc., reflecting information such as the current health level and growth trend of Sesbania cannabina plants.

[0169] In the embodiment of the present invention, when the real-time growth state monitoring module 105 performs real-time dynamic monitoring on the Sesbania cannabina growth region by using the Sesbania cannabina growth monitoring strategy to obtain the real-time growth state of Sesbania cannabina plants, it is used for:

[0170] Real-time collecting real-time growth image data of the Sesbania cannabina growth region by using the Sesbania cannabina growth monitoring strategy;

[0171] Analyze the state quantization values corresponding to the real-time growth image data;

[0172] Determine the real-time growth state of the Sesbania cannabina plants according to the analyzed state quantization values.

[0173] Specifically, in the Sesbania cannabina growth monitoring strategy, determine the monitoring frequency, time, area, and monitoring depth. According to the set parameters, use corresponding devices (such as drones, fixed cameras, ground mobile monitoring devices, etc.) to collect real-time images of the Sesbania cannabina growth area. After obtaining the real-time growth image data, these images need to be processed and analyzed to obtain the corresponding state quantization values. Use machine vision algorithms, image processing techniques, etc. to extract multi-modal plant characteristics (such as morphology, color, texture, etc.) from the images. After completing the analysis of the state quantization values, according to the pre-set rules or standards, convert the analysis results into a description of the real-time growth state of the Sesbania cannabina plants. For example, when the comprehensive state quantization value is within a certain higher range, it is determined that the real-time growth state of the Sesbania cannabina plants is good; when the state quantization value is in the medium range, the growth state is considered average; when the state quantization value is low, the growth state is determined to be poor.

[0174] In the embodiment of the present invention, the Sesbania cannabina growth area is divided into two parts to construct a distributed growth tree of Sesbania cannabina, thereby realizing refined monitoring and performing targeted analysis according to the characteristics of different local areas; collect multi-modal Sesbania cannabina plant images according to the rotation angle, covering multi-dimensional information of Sesbania cannabina at different angles, and overcoming the limitations of a single perspective and data type; reconstruct the Sesbania cannabina plants according to the time-series images to obtain a plant reconstruction image that more comprehensively and accurately reflects the true morphology of Sesbania cannabina; analyze the local growth state of the Sesbania cannabina plants in the node growth area based on multi-modal plant characteristics, and be able to analyze the health status, growth rate, etc. of Sesbania cannabina in each small area, and then determine the global growth state based on the local growth state, taking into account both the overall trend and local differences; dynamically generate the Sesbania cannabina growth monitoring strategy through local and global growth states, enabling the monitoring strategy to be adjusted in real time according to the actual growth situation of Sesbania cannabina, ensuring the timeliness and accuracy of monitoring. Therefore, the real-time monitoring system and method for the growth state of Sesbania cannabina based on machine vision proposed by the present invention can solve the problem of low accuracy in monitoring the growth state of plants.

[0175] Refer to Figure 2 As shown, it is a schematic flowchart of the operation method of the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision provided by an embodiment of the present invention. In this embodiment, the operation method of the real-time monitoring system for the growth state of Sesbania cannabina based on machine vision includes:

[0176] S1. Double - divide the pre - obtained Sesbania cannabina growth area to obtain a Sesbania cannabina distributed growth tree, and extract the node growth area corresponding to each leaf node in the Sesbania cannabina distributed growth tree;

[0177] S2. Collect multi - modal Sesbania cannabina plant images corresponding to the node growth area at a preset rotation angle, and generate Sesbania cannabina time - series images based on the multi - modal Sesbania cannabina plant images;

[0178] S3. Reconstruct the Sesbania cannabina plants in the node growth area according to the Sesbania cannabina time - series images to obtain plant reconstruction images, and extract multi - modal plant features corresponding to the plant reconstruction images using a preset machine vision algorithm;

[0179] S4. Analyze the local growth state of the Sesbania cannabina plants in the node growth area according to the multi - modal plant features, and determine the global growth state of the Sesbania cannabina growth area according to the local growth state;

[0180] S5. Dynamically generate a Sesbania cannabina growth monitoring strategy through the local growth state and the global growth state, and use the Sesbania cannabina growth monitoring strategy to perform real - time dynamic monitoring on the Sesbania cannabina growth area to obtain the real - time growth state of the Sesbania cannabina plants.

[0181] In several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation.

[0182] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above - integrated units can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0184] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above - described exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0185] Therefore, in any regard, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is not limited only by the above description. Thus, it is intended to encompass all changes within the meaning and scope of equivalent elements falling within the scope of protection in the present invention.

[0186] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. A plurality of units or systems stated in the system claims can also be implemented by one unit or system through software or hardware. Words such as first, second, etc. are used to denote names and do not denote any particular order.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A real-time monitoring system for the growth status of Sesbania sesbania based on machine vision, characterized in that: The system comprises: A Sesbania distributed growth tree generation module is used to preliminarily refine the Sesbania growth area obtained in advance to obtain a first re-growth area, locally subdivide the first re-growth area to obtain a second re-growth area, generate a Sesbania distributed growth tree according to the hierarchical relationship and belonging relationship between the first re-growth area and the second re-growth area, and extract the node growth area corresponding to each leaf node in the Sesbania distributed growth tree; A sesbania time series image generation module, used for collecting multimodal sesbania plant images corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant images; a multimodal plant feature extraction module, for reconstructing the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extracting multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm, wherein the multimodal plant features include a visible light image, an infrared image, and a depth image; a global growth state analysis module, configured to analyze the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, extract a state mean corresponding to the local growth state, calculate a global growth state quantization value of the sesbania growth area according to the state mean and a preset state value correction coefficient, and determine the global growth state of the sesbania growth area according to the global growth state quantization value; The real-time growth status monitoring module is used to dynamically generate a monitoring frequency, a monitoring time and a monitoring depth through the local growth status and the global growth status, generate a sesbania growth monitoring strategy according to the monitoring frequency, the monitoring time and the monitoring depth, and use the sesbania growth monitoring strategy to perform real-time dynamic monitoring of the sesbania growth area to obtain the real-time growth status of the sesbania plant.

2. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the sesbania time series image generation module collects the multimodal sesbania plant image corresponding to the node growth area according to the preset rotation angle, it is used to: Determine the initial collection angle and angle step of the Sesbania sesbania plants in the node growth area according to the rotation angle; Determine a collection point according to the initial collection angle and the angle step; Collecting the visible light image, infrared image and depth image corresponding to the Sesbania sesbania plant in the node growth area through the collection point; The visible light image, the infrared image and the depth image are fused into a multimodal Sesbania sesbania plant image corresponding to the node growth area.

3. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When generating the sesbania time series image according to the multimodal sesbania plant image, the sesbania time series image generating module is used to: Extracting the time point corresponding to the multimodal sesbania plant image; Arranging the multimodal sesbania plant images corresponding to the time points in order from early to late; The arranged multimodal Sesbania plant images are used as Sesbania time series images.

4. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the multimodal plant feature extraction module reconstructs the sesbania plants in the node growth area according to the sesbania time series image to obtain the plant reconstructed image, it is used to: Extracting multimodal sesbania plant images at different angles at the same time from the sesbania time series images, and extracting visible light images at different angles from the multimodal sesbania plant images; Extracting feature points of visible light images at different angles one by one, and constructing image associations of visible light images at different angles according to the feature points; According to the image association, matching feature points are identified, and the three-dimensional coordinates of the sesbania plant are calculated according to the point coordinates of the matching feature points, wherein the three-dimensional coordinate calculation formula is: ； in, is the three-dimensional coordinate, is the first scaling factor, is the second scale factor, is the camera intrinsic parameter matrix, are visible light images at different angles. represents the zero matrix, is the rotation matrix, is the translation vector, For the The point coordinates of the matching feature points, For the The point coordinates of the matching feature points; generating point cloud data of the sesbania plant according to the three-dimensional coordinates, generating a triangular mesh of the sesbania plant according to the point cloud data, mapping texture information of the visible light image to the triangular mesh, and obtaining a visible light reconstructed image corresponding to the sesbania plant; reconstructing an infrared image and a depth image in the multimodal sesbania plant image; A plant reconstructed image is determined according to the reconstructed infrared image, the reconstructed depth image and the visible light reconstructed image.

5. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the multimodal plant feature extraction module extracts the multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm, it is used to: The morphological features of the plant reconstructed image are extracted using a machine vision algorithm, wherein the morphological features include plant height features, leaf area features, and stem thickness features: Extracting color features and texture features of the plant reconstructed image; The morphological features, the color features and the texture features are determined as multimodal plant features.

6. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When analyzing the local growth state of the Sesbania sesbania plant in the node growth area according to the multimodal plant characteristics, the global growth state analysis module is used to: Comparing the morphological features in the multimodal plant features with the pre-acquired target morphological features to obtain a first contrast factor, and converting the first contrast factor into a first state value; Comparing the color feature in the multimodal plant feature with the pre-acquired target color feature to obtain a second contrast factor, and converting the second contrast factor into a second state value; Comparing the texture feature in the multimodal plant feature with the pre-acquired target texture feature to obtain a third contrast factor, and converting the third contrast factor into a third state value; The state mean values ​​corresponding to the first state value, the second state value and the third state value are calculated, and the local growth state of the Sesbania sesbania plants in the node growth area is determined according to the state mean values.

7. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the real-time growth status monitoring module uses the sesbania growth monitoring strategy to perform real-time dynamic monitoring on the sesbania growth area to obtain the real-time growth status of the sesbania plant, it is used to: Using the sesbania growth monitoring strategy to collect real-time growth image data of the sesbania growth area in real time; Analyzing the state quantization value corresponding to the real-time growth image data; The real-time growth status of the Sesbania sesbania plant is determined according to the analyzed state quantification value.

8. A method for operating a real-time monitoring system for the growth status of Sesbania sesbania based on machine vision, characterized in that: Used to implement the real-time monitoring system for the growth status of Sesbania sesbania based on machine vision according to any one of claims 1 to 7, the method comprising: Preliminarily refine the pre-acquired sesbania growth region to obtain a first re-growth region, locally subdivide the first re-growth region to obtain a second re-growth region, generate a sesbania distributed growth tree according to the hierarchical relationship and belonging relationship between the first re-growth region and the second re-growth region, and extract the node growth region corresponding to each leaf node in the sesbania distributed growth tree; Collecting a multimodal sesbania plant image corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant image; Reconstructing the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extracting multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm, wherein the multimodal plant features include a visible light image, an infrared image, and a depth image; Analyzing the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, extracting the state mean corresponding to the local growth state, calculating the global growth state quantization value of the sesbania growth area according to the state mean and a preset state value correction coefficient, and determining the global growth state of the sesbania growth area according to the global growth state quantization value; The monitoring frequency, monitoring time and monitoring depth are dynamically generated through the local growth status and the global growth status, a sesbania growth monitoring strategy is generated according to the monitoring frequency, monitoring time and monitoring depth, and the sesbania growth monitoring strategy is used to perform real-time dynamic monitoring of the sesbania growth area to obtain the real-time growth status of the sesbania plant.

Images