Sesbania growth state real-time monitoring system and method based on machine vision

Abstract

The invention relates to the technical field of machine vision, and discloses a sesbania growth state real-time monitoring system and method based on machine vision. The system comprises a sesbania distributed growth tree generation module, a sesbania time sequence image generation module, a multi-modal plant feature extraction module, a global growth state analysis module and a real-time growth state monitoring module, a sesbania growth region is doubly divided to obtain a distributed growth tree, and a region corresponding to a leaf node is extracted; collecting a multi-modal image according to the angle and generating a time sequence image; reconstructing a plant image according to the time sequence image, and extracting multi-modal features of the reconstructed plant image; analyzing a local growth state according to the multi-modal features, and determining a global state according to the local growth state; a monitoring strategy is dynamically generated through the local growth state and the global growth state, and real-time dynamic monitoring of the sesbania growth area is achieved. According to the invention, the accuracy of plant growth state monitoring can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a system and method for real-time monitoring of the growth status of sesbania based on machine vision. Background Art

[0002] With the rapid development of computer technology, computer technology has gradually been applied to the agricultural field. Among them, machine vision technology, as a combination of contemporary computer technology and image sensor network technologies, has begun to be applied to the field of crop growth analysis. It can provide decision support for judging the growth status of crops in the seedling stage. However, in order to be able to analyze the plant growth status more accurately, it is necessary to comprehensively analyze the plant growth area.

[0003] The existing real-time monitoring technology of plant growth status obtains a single plant image through a camera or other image acquisition device to determine the growth status. In practical applications, it is difficult to accurately analyze the growth status of plants by only considering a single plant image, resulting in a relatively single monitoring of the plant growth status, and a lack of comprehensive monitoring of the plant growth status, resulting in low accuracy in monitoring the plant growth status. Summary of the invention

[0004] The present invention provides a machine vision-based real-time monitoring system and method for the growth status of sesbania, the main purpose of which is to solve the problem of low accuracy when monitoring the growth status of plants.

[0005] To achieve the above object, the present invention provides a real-time monitoring system for the growth status of Sesbania sesbania based on machine vision, comprising: A Sesbania distributed growth tree generation module is used to perform a double division on the Sesbania growth area acquired in advance to obtain a Sesbania distributed growth tree, and extract a node growth area corresponding to each leaf node in the Sesbania distributed growth tree; A sesbania time series image generation module, used for collecting multimodal sesbania plant images corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant images; A multimodal plant feature extraction module, used to reconstruct the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extract multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm; A global growth state analysis module, used for analyzing the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, and determining the global growth state of the sesbania growth area according to the local growth state; The real-time growth status monitoring module is used to dynamically generate a sesbania growth monitoring strategy based on the local growth status and the global growth status, and use the sesbania growth monitoring strategy to perform real-time dynamic monitoring of the sesbania growth area to obtain the real-time growth status of the sesbania plant.

[0006] Optionally, when the Sesbania sesbania distributed growth tree generation module performs a double division on the pre-acquired Sesbania sesbania growth area to obtain the Sesbania sesbania distributed growth tree, it is used to: Identify the total area of ​​a first region of the Sesbania sesbania growth region obtained in advance, and extract the first highest digit value corresponding to the area value of the total area of ​​the first region; Performing a first redivision on the sesbania growth region according to the first highest bit value to obtain a first redivision growth region; Identify a second region total area of ​​the first re-growing region, and extract a second highest digit value corresponding to an area value of the second region total area; Performing a second redivision on the first re-growth region according to the second highest bit value to obtain a second re-growth region; Taking the sesbania growing region, the first regrowing region and the second regrowing region as region nodes, and constructing a region path between the sesbania growing region, the first regrowing region and the second regrowing region; A Sesbania sesbania distributed growth tree is generated according to the regional nodes and the regional paths.

[0007] Optionally, when the sesbania time series image generation module collects the multimodal sesbania plant image corresponding to the node growth area according to a preset rotation angle, it is used to: Determine the initial collection angle and angle step of the Sesbania sesbania plants in the node growth area according to the rotation angle; Determine a collection point according to the initial collection angle and the angle step; Collecting the visible light image, infrared image and depth image corresponding to the Sesbania sesbania plant in the node growth area through the collection point; The visible light image, the infrared image and the depth image are fused into a multimodal sesbania plant image corresponding to the node growth area, wherein the multimodal sesbania plant image is: ; in, For the The next collection point Pixels of the moment Multimodal images of Sesbania sesbania plants composed of For the The next collection point Pixels of the moment The visible light image, For the The next collection point Pixels of the moment Composition of infrared images; For the The next collection point Pixels of the moment The depth image composed of.

[0008] Optionally, when generating the sesbania time series image according to the multimodal sesbania plant image, the sesbania time series image generating module is used to: Extracting the time point corresponding to the multimodal Sesbania sesbania plant image; Arranging the multimodal sesbania plant images corresponding to the time points in order from early to late; The arranged multimodal Sesbania plant images are used as Sesbania time series images.

[0009] Optionally, when the multimodal plant feature extraction module reconstructs the Sesbania sesbania plants in the node growth area according to the Sesbania sesbania time series image to obtain the plant reconstructed image, it is used to: Extracting multimodal sesbania plant images at different angles at the same time from the sesbania time series images, and extracting visible light images at different angles from the multimodal sesbania plant images; Extracting feature points of visible light images at different angles one by one, and constructing image associations of visible light images at different angles according to the feature points; According to the image association, matching feature points are identified, and the three-dimensional coordinates of the sesbania plant are calculated according to the point coordinates of the matching feature points, wherein the three-dimensional coordinate calculation formula is: ; in, is the three-dimensional coordinate, is the first scaling factor, is the second scale factor, is the camera intrinsic parameter matrix, are visible light images at different angles. represents the zero matrix, is the rotation matrix, is the translation vector, For the The point coordinates of the matching feature points, For the The point coordinates of the matching feature points; generating point cloud data of the sesbania plant according to the three-dimensional coordinates, generating a triangular mesh of the sesbania plant according to the point cloud data, mapping texture information of the visible light image to the triangular mesh, and obtaining a visible light reconstructed image corresponding to the sesbania plant; reconstructing an infrared image and a depth image in the multimodal sesbania plant image; A plant reconstructed image is determined according to the reconstructed infrared image, the reconstructed depth image and the visible light reconstructed image.

[0010] Optionally, when the multimodal plant feature extraction module extracts the multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm, it is used to: The morphological features of the plant reconstructed image are extracted using a machine vision algorithm, wherein the morphological features include plant height features, leaf area features, and stem thickness features: Extracting color features and texture features of the plant reconstructed image; The morphological features, the color features and the texture features are determined as multimodal plant features.

[0011] Optionally, when analyzing the local growth state of the Sesbania sesbania plant in the node growth area according to the multimodal plant characteristics, the global growth state analysis module is used to: Comparing the morphological features in the multimodal plant features with the pre-acquired target morphological features to obtain a first contrast factor, and converting the first contrast factor into a first state value; Comparing the color feature in the multimodal plant feature with the pre-acquired target color feature to obtain a second contrast factor, and converting the second contrast factor into a second state value; Comparing the texture feature in the multimodal plant feature with the pre-acquired target texture feature to obtain a third contrast factor, and converting the third contrast factor into a third state value; The state mean values ​​corresponding to the first state value, the second state value and the third state value are calculated, and the local growth state of the Sesbania sesbania plants in the node growth area is determined according to the state mean values.

[0012] Optionally, when dynamically generating a sesbania growth monitoring strategy through the local growth status and the global growth status, the real-time growth status monitoring module is used to: determining a first monitoring frequency and a monitoring time of a first re-growth region according to the global growth state; Determine a monitoring area of ​​a second overgrowth area according to the local growth state, and configure a second monitoring frequency and a monitoring depth of the monitoring area; generating a sesbania growth monitoring strategy for the first regrowing area according to the first monitoring frequency and the monitoring time; A sesbania growth monitoring strategy for the second regrowing area is generated according to the second monitoring frequency and the monitoring depth.

[0013] Optionally, when the real-time growth status monitoring module uses the sesbania growth monitoring strategy to perform real-time dynamic monitoring on the sesbania growth area to obtain the real-time growth status of the sesbania plant, it is used to: Using the sesbania growth monitoring strategy to collect real-time growth image data of the sesbania growth area in real time; Analyzing the state quantization value corresponding to the real-time growth image data; The real-time growth status of the Sesbania sesbania plant is determined according to the analyzed state quantification value.

[0014] In order to solve the above problems, the present invention also provides a method for real-time monitoring of the growth status of Sesbania sesbania based on machine vision, the method comprising: Performing double division on the pre-acquired sesbania growth region to obtain a sesbania distributed growth tree, and extracting a node growth region corresponding to each leaf node in the sesbania distributed growth tree; Collecting a multimodal sesbania plant image corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant image; Reconstructing the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extracting multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm; Analyzing the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, and determining the global growth state of the sesbania growth area according to the local growth state; A sesbania growth monitoring strategy is dynamically generated through the local growth state and the global growth state, and the sesbania growth monitoring strategy is used to perform real-time dynamic monitoring on the sesbania growth area to obtain the real-time growth state of the sesbania plant.

[0015] The embodiment of the present invention performs a double division of the sesbania growth area, constructs a sesbania distributed growth tree, and then realizes refined monitoring, and conducts targeted analysis on the characteristics of different local areas; collects multimodal sesbania plant images according to the rotation angle, covers the multidimensional information of sesbania at different angles, and overcomes the limitations of a single perspective and data type; reconstructs the sesbania plant according to the time series image, and obtains a plant reconstruction image that more comprehensively and accurately reflects the true form of sesbania; based on the multimodal plant feature analysis node growth area sesbania plant local growth state, can analyze the health status of each small area sesbania, growth rate, etc., and then determine the global growth state based on the local growth state, considering both the overall trend and the local differences; dynamically generates a sesbania growth monitoring strategy through local and global growth states, so that the monitoring strategy can be adjusted in real time according to the actual growth of sesbania, ensuring the timeliness and accuracy of monitoring. Therefore, the sesbania growth state real-time monitoring system and method based on machine vision proposed by the present invention can solve the problem of low accuracy when monitoring the growth state of plants. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A functional module diagram of a real-time monitoring system for the growth status of Sesbania sesbania based on machine vision provided by an embodiment of the present invention; Figure 2 A schematic flow chart of an operating method of a real-time monitoring system for the growth status of Sesbania sesbania based on machine vision provided in one embodiment of the present invention.

[0017] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0021] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0022] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0023] In fact, the server-side device deployed by the real-time monitoring system for the growth status of sesbania based on machine vision may be composed of one or more devices. The above-mentioned real-time monitoring system for the growth status of sesbania based on machine vision can be implemented as: a business instance, a virtual machine, and a hardware device. For example, the real-time monitoring system for the growth status of sesbania based on machine vision can be implemented as a business instance deployed on one or more devices in a cloud node. In simple terms, the real-time monitoring system for the growth status of sesbania based on machine vision can be understood as a software deployed on a cloud node, which is used to provide a real-time monitoring system for the growth status of sesbania based on machine vision to each user terminal. Alternatively, the real-time monitoring system for the growth status of sesbania based on machine vision can also be implemented as a virtual machine deployed on one or more devices in a cloud node. Application software for managing each user terminal is installed in the virtual machine. Alternatively, the real-time monitoring system for the growth status of sesbania based on machine vision can also be implemented as a server consisting of many hardware devices of the same or different types, and one or more hardware devices are set to provide a real-time monitoring system for the growth status of sesbania based on machine vision to each user terminal.

[0024] In terms of implementation, the real-time monitoring system for the growth status of sesbania based on machine vision and the user end are adapted to each other. That is, the real-time monitoring system for the growth status of sesbania based on machine vision is an application installed on a cloud service platform, and the user end is a client that establishes a communication connection with the application; or the real-time monitoring system for the growth status of sesbania based on machine vision is implemented as a website, and the user end is implemented as a web page; or the real-time monitoring system for the growth status of sesbania based on machine vision is implemented as a cloud service platform, and the user end is implemented as a small program in an instant messaging application.

[0025] Reference Figure 1 , which is a functional module diagram of a machine vision-based real-time monitoring system for the growth status of Sesbania sesbania provided in one embodiment of the present invention.

[0026] The real-time monitoring system 100 for the growth status of Sesbania sesbania based on machine vision of the present invention can be set in a cloud server. In terms of implementation, it can be used as one or more service devices, or it can be installed as an application on the cloud (such as a server of a mobile service operator, a server cluster, etc.), or it can also be developed as a website. According to the functions implemented, the real-time monitoring system 100 for the growth status of Sesbania sesbania based on machine vision can include a Sesbania sesbania distributed growth tree generation module 101, a Sesbania sesbania time series image generation module 102, a multimodal plant feature extraction module 103, a global growth status analysis module 104 and a real-time growth status monitoring module 105. The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by a device processor and can complete fixed functions, which are stored in the memory of the device.

[0027] In the embodiment of the present invention, in the real-time monitoring system of the growth state of sesbania based on machine vision, each of the above modules can be independently implemented and called with other modules. The call here can be understood as that a certain module can connect multiple modules of another type and provide corresponding services for the multiple modules connected to it. For example, the sharing evaluation module can call the same information acquisition module to obtain the information collected by the information acquisition module. Based on the above characteristics, in the real-time monitoring system of the growth state of sesbania based on machine vision provided by the embodiment of the present invention, there is no need to modify the program code, and the scope of application of the real-time monitoring system architecture of the growth state of sesbania based on machine vision can be adjusted by adding modules and directly calling, so as to achieve cluster-type horizontal expansion, so as to achieve the purpose of quickly and flexibly expanding the real-time monitoring system of the growth state of sesbania based on machine vision. In practical applications, the above modules can be arranged in the same device or different devices, or they can be arranged in virtual devices, such as service instances in cloud servers.

[0028] In the following, in conjunction with specific embodiments, each component and specific workflow of the real-time monitoring system for the growth status of Sesbania sesbania based on machine vision are described respectively: The Sesbania distributed growth tree generation module 101 is used to perform a double partition on the pre-acquired Sesbania growth region to obtain a Sesbania distributed growth tree, and extract the node growth region corresponding to each leaf node in the Sesbania distributed growth tree.

[0029] In an embodiment of the present invention, the sesbania distributed growth tree is a tree structure, with the sesbania growth area as the root node, the first growth area as the first-level child node, and the second growth area as the second-level child node. Each node is connected by a regional path, which clearly shows the hierarchical division structure of the sesbania growth area, helps to systematically manage and monitor the sesbania growth areas at different levels, and facilitates the rapid positioning and analysis of the growth status of sesbania in a specific area.

[0030] In the embodiment of the present invention, when the Sesbania sesbania distributed growth tree generation module 101 performs a double division on the pre-acquired Sesbania sesbania growth area to obtain the Sesbania sesbania distributed growth tree, it is used to: Identify the total area of ​​a first region of the Sesbania sesbania growth region obtained in advance, and extract the first highest digit value corresponding to the area value of the total area of ​​the first region; Performing a first redivision on the sesbania growth region according to the first highest bit value to obtain a first redivision growth region; Identify a second region total area of ​​the first re-growing region, and extract a second highest digit value corresponding to an area value of the second region total area; Performing a second redivision on the first re-growth region according to the second highest bit value to obtain a second re-growth region; Taking the sesbania growing region, the first regrowing region and the second regrowing region as region nodes, and constructing a region path between the sesbania growing region, the first regrowing region and the second regrowing region; A Sesbania sesbania distributed growth tree is generated according to the regional nodes and the regional paths.

[0031] In detail, the total area of ​​the pre-acquired sesbania growth area, that is, the total area of ​​the first area, is identified. Satellite remote sensing images can be used to obtain images of the sesbania growth area. Image processing software, combined with GIS technology, can be used to accurately calculate the area of ​​the area based on the coordinate information and scale on the image. For example, if the total area of ​​the first area is 5000, the highest digit value corresponding to the area value of the total area of ​​the first area is extracted, that is, the first highest digit value, and the highest digit value of 5000 is 5. The highest digit value will be used as the basis for the number of shares for the first division. According to the first highest digit value of 5, the sesbania growth area is divided into equal areas to obtain 5 first growth areas, that is, the large area of ​​sesbania growth area is preliminarily refined, which is convenient for subsequent more accurate monitoring of different local areas and helps to discover differences in sesbania growth on a larger scale.

[0032] Specifically, for each area obtained by the first redivision, the total area of ​​its second area is identified. Assuming that the area of ​​one of the first redivision areas is 400, the second highest digit value 4 corresponding to its area value is extracted. This value will be used for the second redivision of the first redivision area. According to the second highest digit value 4, the corresponding first redivision area is divided into 4 parts to obtain the second redivision area. The Sesbania sesbania growth area is further subdivided, so that monitoring can go deep into a smaller local area and capture subtle changes in the growth of Sesbania sesbania.

[0033] Exemplarily, the sesbania growing region is , the first growth region is , the second growth region is , etc., and then the sesbania growing area As the root node, the first growth area As a first-level child node, the second growth area , As secondary child nodes, and clarify the hierarchical relationship and ownership relationship between them, for example Include , Also includes , thus generating a distributed growth tree of Sesbania sesbania.

[0034] Furthermore, the node growth region corresponding to each leaf node in the Sesbania distributed growth tree is extracted. Then, starting from the root node of the Sesbania distributed growth tree, the search is continued downward along the branches of the tree until a leaf node is reached. When the leaf node is traversed, the actual growth region corresponding to the leaf node is determined according to the recorded node information. Since the correspondence between each regional node and the actual Sesbania growth region has been clarified when constructing the Sesbania distributed growth tree, the corresponding node growth region can be found through the name or number of the leaf node, such as the leaf node. When building a tree, the first growth area is After the second division, we can determine the The location and extent of the specific Sesbania growing area represented.

[0035] Furthermore, the Sesbania plant has a three-dimensional structure, and collecting images from a single angle can only obtain partial information, and there is information missing. Therefore, the Sesbania plant should be photographed from multiple perspectives to avoid missing important features due to a single perspective.

[0036] The Sesbania time series image generation module 102 is used to collect the multimodal Sesbania plant images corresponding to the node growth area according to a preset rotation angle, and generate Sesbania time series images according to the multimodal Sesbania plant images.

[0037] In an embodiment of the present invention, the multimodal Sesbania plant image refers to an image obtained by fusing image information about the Sesbania plant acquired from different sensors (such as a visible light camera, an infrared camera, and a depth camera), which includes information about the Sesbania plant in multiple modalities including visible light, infrared light, and depth, and can describe the characteristics and status of the Sesbania plant in a more comprehensive and richer manner.

[0038] In the embodiment of the present invention, when the sesbania time series image generation module 102 collects the multimodal sesbania plant image corresponding to the node growth area according to the preset rotation angle, it is used to: Determine the initial collection angle and angle step of the Sesbania sesbania plants in the node growth area according to the rotation angle; Determine a collection point according to the initial collection angle and the angle step; Collecting the visible light image, infrared image and depth image corresponding to the Sesbania sesbania plant in the node growth area through the collection point; The visible light image, the infrared image and the depth image are fused into a multimodal sesbania plant image corresponding to the node growth area, wherein the multimodal sesbania plant image is: ; in, For the The next collection point Pixels of the moment Multimodal images of Sesbania sesbania plants composed of For the first collection point Pixels of the moment The visible light image, For the The next collection point Pixels of the moment Composition of infrared images; For the The next collection point Pixels of the moment The depth image composed of.

[0039] In detail, when collecting images of Sesbania plants in the node growth area, the initial collection angle is determined based on the rotation angle, that is, the angle when starting to collect images. If the images of Sesbania plants are to be collected comprehensively from different angles, the initial collection angle may be set to 0 degrees, and the angle step refers to the angle interval of rotation after each image collection. For example, an image is collected every 30 degrees of rotation to ensure that image information of the Sesbania plant is obtained from multiple angles. The initial collection angle determines the starting direction of the collection, and the angle step specifies the interval of each rotation. Starting from the initial collection angle, a series of collection points can be determined by rotating the angles specified by the angle step in sequence. For example, if the initial collection angle is 0 degrees and the angle step is 30 degrees, then the collection points are 0 degrees, 30 degrees, 60 degrees, 90 degrees, etc. in sequence. The collection points represent the positions for collecting images of the Sesbania plants at different angles.

[0040] Specifically, after the collection points are determined, at each collection point, the corresponding equipment (such as visible light camera, infrared camera, depth camera, etc.) is used to collect images of the Sesbania plants in the node growth area. The visible light image can reflect the appearance information of the Sesbania plants, such as color, shape, texture, etc.; the infrared image mainly reflects the temperature distribution on the surface of the plant, which helps to understand the physiological state of the plant; the depth image provides three-dimensional spatial information about the plant, such as the distance and depth of each part of the plant, so that for each collection point in the The images collected at all times are fused with visible light images, infrared images and depth images. The multimodal Sesbania plant image is a collection that contains image information of three different modalities. Therefore, by utilizing image information of different modalities, more in-depth Sesbania plant information that cannot be provided by a single modality image can be obtained.

[0041] Furthermore, the growth of Sesbania sesbania is a dynamic process that changes over time. Multimodal Sesbania plant images can provide rich information at a certain moment, but a single image cannot intuitively reflect the continuity and change trend of growth. Therefore, it is necessary to analyze the duration of Sesbania sesbania growth.

[0042] In an embodiment of the present invention, the Sesbania time series image is a multimodal Sesbania plant image arranged in chronological order, representing the corresponding multimodal plant images at different times, recording the state information of the Sesbania plant at different time points. Through the Sesbania time series image, one can intuitively see the changes in the morphology, color, texture, etc. of the Sesbania plant over time, as well as the characteristic changes under different modalities (visible light, infrared, depth, etc.).

[0043] In the embodiment of the present invention, when generating the sesbania time series image according to the multimodal sesbania plant image, the sesbania time series image generating module 102 is used to: Extracting the time point corresponding to the multimodal Sesbania sesbania plant image; Arranging the multimodal sesbania plant images corresponding to the time points in order from early to late; The arranged multimodal Sesbania plant images are used as Sesbania time series images.

[0044] In detail, the multimodal Sesbania plant images are acquired at different times, and each image corresponds to a specific acquisition moment. After extracting the time points of all the multimodal Sesbania plant images, these time points are compared and sorted, and the corresponding multimodal Sesbania plant images are arranged in chronological order, starting from the earliest time point, to ensure the time continuity between the images, so that the changes of the Sesbania plant in the time dimension can be clearly observed later, and the multimodal Sesbania plant images are arranged in chronological order from early to late to form a Sesbania time series image.

[0045] Furthermore, the Sesbania time series images contain multimodal Sesbania plant image information at different time points and multiple angles. In order to more realistically reflect the actual morphology and structure of the plant, it is necessary to reconstruct a three-dimensional model of the Sesbania plant.

[0046] The multimodal plant feature extraction module 103 is used to reconstruct the Sesbania sesbania plants in the node growth area according to the Sesbania sesbania time series image to obtain a plant reconstructed image, and use a preset machine vision algorithm to extract multimodal plant features corresponding to the plant reconstructed image.

[0047] In an embodiment of the present invention, the plant reconstructed image refers to a three-dimensional image containing multi-faceted information of the Sesbania sesbania plant, which is generated by processing and analyzing multimodal images (visible light, infrared, depth, etc.) in the Sesbania sesbania time series images, and can intuitively display the morphology, structure and physiological state of the Sesbania sesbania plant.

[0048] In the embodiment of the present invention, when reconstructing the sesbania plants in the node growth area according to the sesbania time series image to obtain the plant reconstructed image, the multimodal plant feature extraction module 103 is used to: Extracting multimodal sesbania plant images at different angles at the same time from the sesbania time series images, and extracting visible light images at different angles from the multimodal sesbania plant images; Extracting feature points of visible light images at different angles one by one, and constructing image associations of visible light images at different angles according to the feature points; According to the image association, matching feature points are identified, and the three-dimensional coordinates of the sesbania plant are calculated according to the point coordinates of the matching feature points, wherein the three-dimensional coordinate calculation formula is: ; in, is the three-dimensional coordinate, is the first scaling factor, is the second scale factor, is the camera intrinsic parameter matrix, are visible light images at different angles. represents the zero matrix, is the rotation matrix, is the translation vector, For the The point coordinates of the matching feature points, For the The point coordinates of the matching feature points; generating point cloud data of the sesbania plant according to the three-dimensional coordinates, generating a triangular mesh of the sesbania plant according to the point cloud data, mapping texture information of the visible light image to the triangular mesh, and obtaining a visible light reconstructed image corresponding to the sesbania plant; reconstructing an infrared image and a depth image in the multimodal sesbania plant image; A plant reconstructed image is determined according to the reconstructed infrared image, the reconstructed depth image and the visible light reconstructed image.

[0049] Specifically, from the Sesbania time series images, multimodal Sesbania plant images at different angles at the same time are identified, that is, the images at the same time can reflect the state of the plant at that time, and the images at different angles provide multi-viewing information of the plant, thereby extracting visible light images at different angles at the same time, such as Angle at the moment The corresponding visible light image is ,angle The corresponding visible light image is ,angle The corresponding visible light image is And so on, until all visible light images at different angles are extracted, and visible light images at different angles are obtained, and multimodal images of Sesbania sesbania plants at different collection points and times are obtained. , the feature points are extracted by using the scale-invariant feature transformation algorithm. The feature points are obtained by constructing the scale space extreme value detection. For an image , whose scale space is defined as ,in, is the Gaussian kernel function, is the scale factor, Represents the convolution operation, through Gaussian difference functions of different scales To detect feature points, is a constant.

[0050] Specifically, the Euclidean distance is used to match the feature points between visible light images at different angles. and , its Euclidean distance for , then identify the closest feature point pair, establish the connection between images of different perspectives and time, and perform triangulation based on the feature points. Assume that the camera intrinsic parameter matrix is , for the matching feature point pairs, in the visible light image and visible light images , In the figure, the corresponding image point coordinates are and , then the three-dimensional point coordinates satisfy , ,in, and is the scaling factor, is the rotation matrix, is the translation vector, Represents a two-dimensional vector, which is used to project the coordinates of a three-dimensional space point into a specific transformation of a two-dimensional space, where middle Represents the visible light image matrix at different angles, Represents the zero matrix, and the coordinates of the three-dimensional point are obtained by solving the equation system , and then combine the three-dimensional coordinates obtained by triangulation into a point cloud, and use the Delaunay triangulation algorithm to generate a triangular mesh on the optimized point cloud. , Delaunay triangulation makes the circumscribed circle of each triangle contain no other points, and then maps the texture information of the multimodal Sesbania sesbania plant image to the grid model. According to the correspondence between the feature points, the color, texture and other information of the images at different acquisition points and times are accurately fitted to the three-dimensional model surface, and finally the three-dimensional reconstructed image of the Sesbania sesbania plant is obtained.

[0051] Furthermore, the infrared image and depth image in the multimodal image of the Sesbania sesbania plant are reconstructed, and the reconstruction steps are consistent with the reconstruction steps of the visible light image, wherein the infrared image reconstruction may focus more on the accurate restoration of temperature information, and the depth image reconstruction focuses on the precise representation of three-dimensional spatial distance information, so as to determine the reconstructed image of the plant based on the reconstructed infrared image, the reconstructed depth image and the visible light reconstructed image. The reconstructed image of the plant is a three-dimensional image that integrates multimodal information such as visible light, infrared and depth, and it includes various information such as the appearance, temperature distribution and three-dimensional spatial structure of the Sesbania sesbania plant.

[0052] Furthermore, the reconstructed image of the plant contains multimodal information such as visible light, infrared, and depth, which can reflect the characteristics of the Sesbania plant from different angles, integrate various information, and comprehensively evaluate the growth status of the Sesbania plant.

[0053] In an embodiment of the present invention, the multimodal plant characteristics include morphological characteristics, color characteristics and texture characteristics of the sesbania plant. The morphological characteristics describe the external morphology of the sesbania plant through geometric parameters (plant height, leaf area, stem thickness); the color characteristics reflect the health status of the leaves through the color mean; and the texture characteristics describe the microstructure of the leaves through the grayscale co-occurrence matrix.

[0054] In the embodiment of the present invention, when the multimodal plant feature extraction module 103 extracts the multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm, it is used to: The morphological features of the plant reconstructed image are extracted using the following machine vision algorithm: ; ; ; in, For the The plant height characteristics in the morphological characteristics of the moment, for The leaf area characteristics in the morphological characteristics of the moment, For the The stem thickness characteristics in the morphological characteristics of the moment, is the ordinate of the highest point of the Sesbania sesbania plant in the reconstructed image of the plant, is the plant height control factor, is the leaf area control factor, is the controlling factor of stem thickness, Reconstructing the image of the plant The pixel value of the position, is the horizontal coordinate of the Sesbania sesbania plant in the plant reconstructed image, is the width of the reconstructed image of the plant, The length of the reconstructed image for the plant; Extracting color features and texture features of the plant reconstructed image; The morphological features, the color features and the texture features are determined as multimodal plant features.

[0055] Specifically, the morphological characteristics include plant height, leaf area, and stem diameter, so the plant height calculation formula is: , For Sesbania in time The plant height, is the ordinate of the highest point of the sesbania in the image, is the ordinate of the lowest point of the sesbania in the image, that is, the height of the sesbania is calculated by vertical projection, reflecting its growth status. The leaf area calculation formula is: , For Sesbania in time leaf area, For the binary image The pixel value of the position (1 for sesbania, 0 for background), It means to sum all the pixels in the image, that is, to estimate the leaf area and reflect the growth of the leaves by counting the number of pixels of Sesbania in the binary image; the calculation formula of stem thickness is , For Sesbania in time The stem is thick, is the horizontal coordinate of the rightmost point of Sesbania tiana in the image, is the horizontal coordinate of the leftmost point of the sesbania in the image, that is, the stem thickness of the sesbania is calculated by horizontal projection, reflecting the development of the stem, and the control factor in the machine vision algorithm, , and Determines the contribution of plant height, leaf area and stem thickness to morphological characteristics. If you want plant height to dominate the comprehensive morphological characteristics, you can increase it appropriately. If you want to highlight the influence of leaf area characteristics, you can increase By adjusting the control factor, the importance of each morphological feature in the comprehensive evaluation can be flexibly adjusted. That is, when calculating the plant height feature in the morphological feature, the plant height control factor Set to 1 to control the leaf area and stem thickness control factors When calculating the leaf area characteristics in the morphological characteristics, the leaf area control factor Set to 1, the plant height control factor and stem thickness control factors Set to 0; when calculating the stem thickness feature in the morphological feature, the stem thickness control factor Set to 1, the plant height control factor and leaf area control factor It was set to 0, thus achieving independent calculation and analysis of the three morphological characteristics of plant height, leaf area and stem diameter.

[0056] Specifically, the color feature refers to the mean color of the leaves, and the mean color of the leaves is calculated as , For Sesbania in time The average leaf color of For the image The color value of the position (in RGB or HSV color space), is the total number of pixels of Sesbania sesbania in the image, The sum of all pixels in the image, that is, by calculating the average color of the leaves, reflects the health of the leaves (such as whether they are yellowing or withered), and the texture feature is obtained through the gray level co-occurrence matrix, that is , is the gray-level co-occurrence matrix The value of the position, indicating the gray level and The frequency of co-occurrence in a particular direction, For the image The gray value of the position, Indicator function, the value is 1 when the condition in the brackets is met, otherwise it is 0, that is, That is, by counting the frequency of grayscale co-occurrence, the texture characteristics of Sesbania sesbania leaves are extracted to reflect the roughness and uniformity of the leaves, thereby determining the morphological characteristics, color characteristics and texture characteristics as multimodal plant characteristics.

[0057] Furthermore, the multimodal plant characteristics cover various information such as the plant's morphology, color, texture, temperature, spatial structure, etc. The comprehensive multimodal characteristics can comprehensively and meticulously reflect the actual growth status of the Sesbania plant in the node growth area, avoiding the limitations of single feature analysis.

[0058] The global growth state analysis module 104 is used to analyze the local growth state of the Sesbania plants in the node growth area according to the multimodal plant characteristics, and determine the global growth state of the Sesbania growth area according to the local growth state.

[0059] In the embodiment of the present invention, the local growth state refers to the growth state of the Sesbania sesbania plant in the node growth area corresponding to the leaf node.

[0060] In the embodiment of the present invention, when analyzing the local growth state of the Sesbania sesbania plant in the node growth area according to the multimodal plant characteristics, the global growth state analysis module 104 is used to: Comparing the morphological features in the multimodal plant features with the pre-acquired target morphological features to obtain a first contrast factor, and converting the first contrast factor into a first state value; Comparing the color feature in the multimodal plant feature with the pre-acquired target color feature to obtain a second contrast factor, and converting the second contrast factor into a second state value; Comparing the texture feature in the multimodal plant feature with the pre-acquired target texture feature to obtain a third contrast factor, and converting the third contrast factor into a third state value; The state mean values ​​corresponding to the first state value, the second state value and the third state value are calculated, and the local growth state of the Sesbania sesbania plants in the node growth area is determined according to the state mean values.

[0061] In detail, the morphological features in the multimodal plant features include the plant height, stem thickness, and leaf area of ​​the Sesbania plant, and the pre-acquired target morphological features are the morphological standards that the Sesbania plant should have in the growth area of ​​the node under an ideal state. The actually extracted morphological features are compared with the target morphological features, and the degree of difference between the two is calculated to obtain the first comparison factor. For example, if the target plant height is 50 cm and the actually measured plant height is 40 cm, then there is a difference in the plant height indicator. The first comparison factor is represented by the difference between the two, and then the first comparison factor is converted into a first state value. If the difference of the first comparison factor is less than a preset threshold value, the first state value is marked as 1, otherwise it is marked as 0, that is, if the first comparison factor is small, It means that the actual morphological features are closer to the target morphological features, and the first state value after conversion is higher; conversely, if the difference is large, the first state value is lower; the color features reflect the color information of the leaves, stems and other parts of the field sesbania plant, such as the greenness of the leaves, whether there is yellowing or discoloration, etc. The target color features are also pre-set ideal color standards. The actual color features are compared with the target color features, and the second contrast factor is obtained by analyzing the difference in the leaf color mean parameters. The second contrast factor is then converted into the second state value according to the threshold conversion rule, so that it represents the degree of conformity of the color features within a unified numerical range. The threshold conversion rule is that when the difference in the contrast factor is less than the preset threshold, the state value is marked as 1, otherwise it is marked as 0.

[0062] Specifically, the texture feature reflects the texture details on the surface of the Sesbania sesbania plant, such as the clarity of the leaf lines, whether there are damages or spots, etc.; the target texture feature is the texture standard under ideal conditions. The actual texture feature and the target texture feature are compared to obtain a third contrast factor, and the third contrast factor is converted into a third state value according to the threshold conversion rule to measure the state of the texture feature, and the state mean corresponding to the first state value, the second state value and the third state value is calculated. The local growth state of the Sesbania sesbania plant in the node growth area is determined according to this state mean. If the state mean is a value of 1, it means that the local growth state is a healthy state. If the state mean is 2 / 3, the local growth state is a sub-healthy state. If the state mean is 1 / 3, the local growth state is a poor state. If the state mean is 0, the local growth state is a seriously unhealthy state.

[0063] Furthermore, the sesbania growth area is composed of multiple node growth areas. The local growth state of each node growth area is a specific reflection of the growth situation of sesbania in that specific area. Combining local information can help us understand the situation of the entire growth area.

[0064] In the embodiment of the present invention, the global growth state refers to the sesbania growth state of the sesbania growth area corresponding to the root node in the sesbania distributed growth tree determined according to the local growth state of the leaf nodes in the sesbania distributed growth tree.

[0065] In the embodiment of the present invention, when determining the global growth state of the sesbania growth area according to the local growth state, the global growth state analysis module 104 is used to: Extracting a state mean corresponding to the local growth state; The global growth state quantization value of the Sesbania sesbania growth area is calculated according to the state mean: ; in, is the quantized value of the global growth state, The first growth region Second growth zone The corresponding state mean, The first growth region Second growth zone The corresponding state value correction coefficient, is the total number of multimodal plant features, is the number of regions in the first growth region, The first growth region The number of regions in the corresponding second growth region; The global growth state of the sesbania growth area is determined according to the global growth state quantification value.

[0066] In detail, the local state mean corresponding to the second growth area as a whole is calculated based on the local growth state of the Sesbania plants in each node growth area (i.e., the second growth area). For example, each leaf node in the second growth area corresponds to a state value, and the state values ​​corresponding to all leaf nodes are averaged to obtain the local state mean. Therefore, the local state mean corresponding to all nodes in the first growth area is obtained based on the leaf nodes, and the global growth state quantization value of the Sesbania growth area is calculated based on the local state mean.

[0067] Specifically, It is to adjust the local state mean. If the state value of the sesbania corresponding to the sesbania growth area corresponding to the leaf node is misjudged, and the state value 1 is judged as the state value 0, or the state value 0 is judged as the state value 1, the state value correction coefficient will be used. and the total number of features, such as The corresponding status value is 1. The corresponding state value is 1 / 3, The corresponding status value is 0. The local state mean is 4 / 9, and After verification, the corresponding state value should be 2 / 3, which means that two of the multimodal plant features are in a healthy state, and one is not. At this time, the state value correction coefficient The value of is 2, and then the global growth state of the sesbania growth area is determined according to the calculated global growth state quantization value. When the global growth state quantization value is 1, it indicates that the global growth state of the sesbania growth area is good. When the global growth state quantization value is 0, it indicates that the global growth state of the sesbania growth area is poor. When the global growth state quantization value is less than 1 and greater than 0, it indicates that the global growth state of the sesbania growth area is average.

[0068] For example, in a tree structure, if the root node is A, the first-level nodes are A1 and A2, the leaf nodes are A11, A12, A13 corresponding to A1, and A21 and A22 corresponding to A2, and each leaf node corresponds to a value, the average of the values ​​of the leaf nodes corresponding to A1 is taken as the value of A1, the average of the values ​​of the leaf nodes corresponding to A2 is taken as the value of A2, and then the average of the values ​​of A1 and A2 is taken as the value of the root node A, then the root node A represents the global growth state of the field sesbania growth area.

[0069] Furthermore, the growth of Sesbania sesbania is a dynamic process. There are differences in the growth conditions at different growth stages and in different local areas. Different local growth states and global growth states require different monitoring methods, thereby improving the pertinence and effectiveness of the monitoring strategy.

[0070] The real-time growth status monitoring module 105 is used to dynamically generate a sesbania growth monitoring strategy based on the local growth status and the global growth status, and use the sesbania growth monitoring strategy to perform real-time dynamic monitoring of the sesbania growth area to obtain the real-time growth status of the sesbania plant.

[0071] In the embodiment of the present invention, the sesbania growth monitoring strategy refers to a series of monitoring plans formulated in order to comprehensively and accurately grasp the growth of sesbania. Through a reasonable monitoring strategy, problems occurring during the growth of sesbania can be discovered in a timely manner, thereby ensuring the healthy growth of sesbania and improving the yield and quality.

[0072] In the embodiment of the present invention, when dynamically generating a sesbania growth monitoring strategy based on the local growth status and the global growth status, the real-time growth status monitoring module 105 is used to: determining a first monitoring frequency and a monitoring time of a first re-growth region according to the global growth state; Determine a monitoring area of ​​a second overgrowth area according to the local growth state, and configure a second monitoring frequency and a monitoring depth of the monitoring area; generating a sesbania growth monitoring strategy for the first regrowing area according to the first monitoring frequency and the monitoring time; A sesbania growth monitoring strategy for the second regrowing area is generated according to the second monitoring frequency and the monitoring depth.

[0073] In detail, the global growth status reflects the comprehensive growth situation of the entire sesbania growth area. If the global growth status is good, it means that the overall growth of sesbania is relatively stable. Then the first monitoring frequency of the first growth area (the larger area obtained by the initial division of the sesbania growth area) can be relatively low. For example, the original plan to monitor once a day can now be adjusted to once every three days. On the contrary, if the global growth status is not good, such as the overall growth rate of sesbania is slow, there are more signs of diseases and pests, etc., it is necessary to increase the monitoring frequency, which may be increased from once a day to multiple times a day, so as to timely grasp the changes in sesbania growth and take corresponding measures; the determination of monitoring time is also related to the global growth status. When the global growth status is good, the monitoring time can choose a regular time period, such as monitoring during the day when there is sufficient light, so as to obtain clear images and data. However, if the global growth status is abnormal, it may be necessary to adjust the monitoring time according to the specific situation. If it is suspected that the sesbania is affected by the low temperature at night, it is necessary to increase monitoring during specific time periods at night to determine the impact of temperature changes on sesbania growth.

[0074] Specifically, the local growth status shows the specific growth conditions of the second growth area (a smaller area obtained by further subdividing the first growth area). By analyzing the local growth status, areas with abnormal growth status are identified as key monitoring areas. For example, if the leaves of the sesbania plants in certain areas turn yellow or wither, these areas are determined as monitoring areas. Therefore, for the determined monitoring areas, the second monitoring frequency is configured according to the degree of abnormality of their local growth status. If the growth status of the area is seriously abnormal, such as an area where pests and diseases break out, the monitoring frequency must be greatly increased, possibly to once an hour; for areas with slightly abnormal growth status, the monitoring frequency can be relatively low, such as twice a day; monitoring depth refers to the level of detail of the sesbania growth information obtained in the monitoring area. For monitoring areas with serious abnormal growth status, the monitoring depth needs to be increased, such as not only monitoring the appearance of the sesbania plants, but also in-depth analysis of soil nutrients. , root growth and other internal factors, thereby combining the first monitoring frequency and monitoring time determined earlier to generate a monitoring strategy for the growth of Sesbania in the first growth area. For example, the first monitoring frequency is once every three days, and the monitoring time is from 9 to 11 a.m. every day. Then the monitoring strategy for this area is to conduct a comprehensive monitoring of the first growth area from 9 to 11 a.m. every three days. The monitoring content includes the use of drones to collect large-area images to obtain information such as the overall growth trend of Sesbania. According to the determined second monitoring frequency and monitoring depth, a monitoring strategy for the growth of Sesbania in the second growth area is generated. For example, the second monitoring frequency is to monitor twice a day, in the morning and afternoon respectively, and the monitoring depth is to comprehensively analyze soil nutrients and plant physiological indicators. Then the monitoring strategy for this area is to conduct detailed monitoring of the determined monitoring area in the morning and afternoon every day, including collecting soil samples for nutrient analysis, and using professional instruments to detect physiological indicators such as the photosynthetic rate and transpiration rate of the plant.

[0075] Furthermore, the growth trend of the Sesbania sesbania plants can be monitored in real time according to the Sesbania sesbania growth monitoring strategy, such as whether the growth rate is accelerating or slowing down, which helps to determine whether the current growth environment is suitable and whether the Sesbania sesbania is developing towards the expected growth target.

[0076] In the embodiment of the present invention, the real-time growth status refers to the actual growth status of the Sesbania plant at the current moment, including the characteristic performances of the Sesbania plant in terms of morphology, color, texture, etc., reflecting the current health level, growth trend and other information of the Sesbania plant.

[0077] In the embodiment of the present invention, when the real-time growth status monitoring module 105 uses the sesbania growth monitoring strategy to perform real-time dynamic monitoring on the sesbania growth area to obtain the real-time growth status of the sesbania plant, it is used to: Using the sesbania growth monitoring strategy to collect real-time growth image data of the sesbania growth area in real time; Analyzing the state quantization value corresponding to the real-time growth image data; The real-time growth status of the Sesbania sesbania plant is determined according to the analyzed state quantification value.

[0078] In detail, the frequency, time, area and depth of monitoring are determined in the sesbania growth monitoring strategy. According to the set parameters, corresponding equipment (such as drones, fixed cameras, ground mobile monitoring equipment, etc.) is used to collect real-time images of the sesbania growth area. After obtaining the real-time growth image data, these images need to be processed and analyzed to obtain corresponding state quantification values. Multimodal plant features (such as morphology, color, texture, etc.) are extracted from the images using machine vision algorithms, image processing techniques, etc. After completing the analysis of the state quantification values, the analysis results are converted into a real-time growth state description of the sesbania plant according to pre-set rules or standards. For example, when the comprehensive state quantification value is within a certain higher range, the real-time growth state of the sesbania plant is judged to be good; when the state quantification value is in a medium range, the growth state is considered to be average; when the state quantification value is low, the growth state is determined to be poor.

[0079] The embodiment of the present invention performs a double division of the sesbania growth area, constructs a sesbania distributed growth tree, and then realizes refined monitoring, and conducts targeted analysis on the characteristics of different local areas; collects multimodal sesbania plant images according to the rotation angle, covers the multidimensional information of sesbania at different angles, and overcomes the limitations of a single perspective and data type; reconstructs the sesbania plant according to the time series image, and obtains a plant reconstruction image that more comprehensively and accurately reflects the true form of sesbania; based on the multimodal plant feature analysis node growth area sesbania plant local growth state, can analyze the health status of each small area sesbania, growth rate, etc., and then determine the global growth state based on the local growth state, considering both the overall trend and the local differences; dynamically generates a sesbania growth monitoring strategy through local and global growth states, so that the monitoring strategy can be adjusted in real time according to the actual growth of sesbania, ensuring the timeliness and accuracy of monitoring. Therefore, the sesbania growth state real-time monitoring system and method based on machine vision proposed by the present invention can solve the problem of low accuracy when monitoring the growth state of plants.

[0080] Reference Figure 2 FIG. 1 is a flow chart of an operation method of a real-time monitoring system for the growth status of sesbania based on machine vision according to an embodiment of the present invention. In this embodiment, the operation method of the real-time monitoring system for the growth status of sesbania based on machine vision includes: S1, performing double division on the pre-acquired sesbania growth area to obtain a sesbania distributed growth tree, and extracting a node growth area corresponding to each leaf node in the sesbania distributed growth tree; S2, collecting a multimodal sesbania plant image corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant image; S3, reconstructing the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extracting multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm; S4, analyzing the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, and determining the global growth state of the sesbania growth area according to the local growth state; S5. Dynamically generate a sesbania growth monitoring strategy based on the local growth status and the global growth status, and use the sesbania growth monitoring strategy to dynamically monitor the sesbania growth area in real time to obtain the real-time growth status of the sesbania plant.

[0081] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0082] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0084] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0085] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is not limited only according to the above description, and it is intended that all changes within the meaning and scope of equivalent elements within the scope of protection are included in the present invention.

[0086] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A real-time monitoring system for the growth status of Sesbania sesbania based on machine vision, characterized in that: The system comprises: A Sesbania distributed growth tree generation module is used to perform a double division on the Sesbania growth area acquired in advance to obtain a Sesbania distributed growth tree, and extract a node growth area corresponding to each leaf node in the Sesbania distributed growth tree; A sesbania time series image generation module, used for collecting multimodal sesbania plant images corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant images; A multimodal plant feature extraction module, used to reconstruct the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extract multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm; A global growth state analysis module, used for analyzing the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, and determining the global growth state of the sesbania growth area according to the local growth state; The real-time growth status monitoring module is used to dynamically generate a sesbania growth monitoring strategy based on the local growth status and the global growth status, and use the sesbania growth monitoring strategy to perform real-time dynamic monitoring of the sesbania growth area to obtain the real-time growth status of the sesbania plant.

2. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: The Sesbania distributed growth tree generation module is used to: Identify the total area of ​​a first region of the Sesbania sesbania growth region obtained in advance, and extract the first highest digit value corresponding to the area value of the total area of ​​the first region; Performing a first redivision on the sesbania growth region according to the first highest bit value to obtain a first redivision growth region; Identify a second region total area of ​​the first re-growing region, and extract a second highest digit value corresponding to an area value of the second region total area; Performing a second redivision on the first re-growth region according to the second highest bit value to obtain a second re-growth region; Taking the sesbania growing region, the first regrowing region and the second regrowing region as region nodes, and constructing a region path between the sesbania growing region, the first regrowing region and the second regrowing region; A Sesbania sesbania distributed growth tree is generated according to the regional nodes and the regional paths.

3. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the sesbania time series image generation module collects the multimodal sesbania plant image corresponding to the node growth area according to the preset rotation angle, it is used to: Determine the initial collection angle and angle step of the Sesbania sesbania plants in the node growth area according to the rotation angle; Determine a collection point according to the initial collection angle and the angle step; Collecting the visible light image, infrared image and depth image corresponding to the Sesbania sesbania plant in the node growth area through the collection point; The visible light image, the infrared image and the depth image are fused into a multimodal sesbania plant image corresponding to the node growth area, wherein the multimodal sesbania plant image is: ； in, For the The next collection point Pixels of the moment Multimodal images of Sesbania sesbania plants composed of For the The next collection point Pixels of the moment The visible light image, For the The next collection point Pixels of the moment Composition of infrared images; For the The next collection point Pixels of the moment The depth image composed of.

4. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When generating the sesbania time series image according to the multimodal sesbania plant image, the sesbania time series image generating module is used to: Extracting the time point corresponding to the multimodal Sesbania sesbania plant image; Arranging the multimodal sesbania plant images corresponding to the time points in order from early to late; The arranged multimodal Sesbania plant images are used as Sesbania time series images.

5. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the multimodal plant feature extraction module reconstructs the sesbania plants in the node growth area according to the sesbania time series image to obtain the plant reconstructed image, it is used to: Extracting multimodal sesbania plant images at different angles at the same time from the sesbania time series images, and extracting visible light images at different angles from the multimodal sesbania plant images; Extracting feature points of visible light images at different angles one by one, and constructing image associations of visible light images at different angles according to the feature points; According to the image association, matching feature points are identified, and the three-dimensional coordinates of the sesbania plant are calculated according to the point coordinates of the matching feature points, wherein the three-dimensional coordinate calculation formula is: ； in, is the three-dimensional coordinate, is the first scaling factor, is the second scale factor, is the camera intrinsic parameter matrix, are visible light images at different angles. represents the zero matrix, is the rotation matrix, is the translation vector, For the The point coordinates of the matching feature points, For the The point coordinates of the matching feature points; generating point cloud data of the sesbania plant according to the three-dimensional coordinates, generating a triangular mesh of the sesbania plant according to the point cloud data, mapping texture information of the visible light image to the triangular mesh, and obtaining a visible light reconstructed image corresponding to the sesbania plant; reconstructing an infrared image and a depth image in the multimodal sesbania plant image; A plant reconstructed image is determined according to the reconstructed infrared image, the reconstructed depth image and the visible light reconstructed image.

6. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the multimodal plant feature extraction module extracts the multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm, it is used to: The morphological features of the plant reconstructed image are extracted using a machine vision algorithm, wherein the morphological features include plant height features, leaf area features, and stem thickness features: Extracting color features and texture features of the plant reconstructed image; The morphological features, the color features and the texture features are determined as multimodal plant features.

7. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When analyzing the local growth state of the Sesbania sesbania plant in the node growth area according to the multimodal plant characteristics, the global growth state analysis module is used to: Comparing the morphological features in the multimodal plant features with the pre-acquired target morphological features to obtain a first contrast factor, and converting the first contrast factor into a first state value; Comparing the color feature in the multimodal plant feature with the pre-acquired target color feature to obtain a second contrast factor, and converting the second contrast factor into a second state value; Comparing the texture feature in the multimodal plant feature with the pre-acquired target texture feature to obtain a third contrast factor, and converting the third contrast factor into a third state value; The state mean values ​​corresponding to the first state value, the second state value and the third state value are calculated, and the local growth state of the Sesbania sesbania plants in the node growth area is determined according to the state mean values.

8. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When dynamically generating a sesbania growth monitoring strategy through the local growth status and the global growth status, the real-time growth status monitoring module is used to: determining a first monitoring frequency and a monitoring time of a first re-growth region according to the global growth state; Determine a monitoring area of ​​a second overgrowth area according to the local growth state, and configure a second monitoring frequency and a monitoring depth of the monitoring area; generating a sesbania growth monitoring strategy for the first regrowing area according to the first monitoring frequency and the monitoring time; A sesbania growth monitoring strategy for the second regrowing area is generated according to the second monitoring frequency and the monitoring depth.

9. The machine vision-based real-time monitoring system for the growth status of Sesbania sesbania according to claim 1, characterized in that: When the real-time growth status monitoring module uses the sesbania growth monitoring strategy to perform real-time dynamic monitoring on the sesbania growth area to obtain the real-time growth status of the sesbania plant, it is used to: Using the sesbania growth monitoring strategy to collect real-time growth image data of the sesbania growth area in real time; Analyzing the state quantization value corresponding to the real-time growth image data; The real-time growth status of the Sesbania sesbania plant is determined according to the analyzed state quantification value.

10. A method for operating a real-time monitoring system for the growth status of Sesbania sesbania based on machine vision, characterized in that: Used to implement the real-time monitoring system for the growth status of Sesbania sesbania based on machine vision according to any one of claims 1 to 9, the method comprising: Performing double division on the pre-acquired sesbania growth region to obtain a sesbania distributed growth tree, and extracting a node growth region corresponding to each leaf node in the sesbania distributed growth tree; Collecting a multimodal sesbania plant image corresponding to the node growth area according to a preset rotation angle, and generating a sesbania time series image according to the multimodal sesbania plant image; Reconstructing the sesbania plants in the node growth area according to the sesbania time series image to obtain a plant reconstructed image, and extracting multimodal plant features corresponding to the plant reconstructed image using a preset machine vision algorithm; Analyzing the local growth state of the sesbania plants in the node growth area according to the multimodal plant characteristics, and determining the global growth state of the sesbania growth area according to the local growth state; A sesbania growth monitoring strategy is dynamically generated through the local growth state and the global growth state, and the sesbania growth monitoring strategy is used to perform real-time dynamic monitoring on the sesbania growth area to obtain the real-time growth state of the sesbania plant.

Images