A method and device for monitoring and identifying vegetation growth status based on image vision

Abstract

The present invention discloses a vegetation growth status monitoring and identification method and device based on image vision, which relates to the field of image processing technology; preprocessing an initial RGB image and an initial thermal infrared image to obtain a target RGB image and a target thermal infrared image; identifying a target interval in the target RGB image, and obtaining a first image and a second image according to the target interval; fusing the first image and the second image to extract features to obtain a first feature image, and enhancing the features of the first feature image to obtain a second feature image; converting the second feature image into a target image through a decoder, extracting features from the target image to obtain target features, and searching for corresponding vegetation status information in a preset database according to the target features. By extracting features from images after feature fusion and feature enhancement, the detail features are retained to the greatest extent, and the vegetation status information in the preset database is searched according to the target features, thereby improving the accuracy and reliability of vegetation health detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and a device for monitoring and identifying vegetation growth status based on image vision. Background Art

[0002] With the acceleration of urbanization, urban landscaping has become increasingly important as an important means to improve the urban ecological environment and beautify the urban landscape. However, the traditional vegetation maintenance and management method is highly dependent on manual inspections, which is not only inefficient and costly, but also limited by human factors, making it difficult to ensure the comprehensiveness and timeliness of monitoring. Manual inspections can often only cover some areas, and it is difficult to detect subtle changes in the growth of vegetation at the first time, such as early symptoms of pests and diseases, signs of nutrient deficiency, or dead branches and leaves. If these problems are not dealt with in a timely manner, they will seriously affect the healthy growth of vegetation and the overall landscape effect.

[0003] In order to solve the problems of low efficiency, high cost, and lack of comprehensiveness and timeliness of monitoring in traditional vegetation maintenance and management methods, existing technologies have begun to collect images through image acquisition equipment, and conduct in-depth analysis of the collected images through image processing technology to identify the growth status of vegetation, including pest and disease detection, nutritional status assessment, etc. However, despite the significant advantages of this technology, in actual applications, the images collected are often interfered by the outside world, and the quality of the collected images is often low. Especially when affected by pests and diseases, it is difficult to timely detect subtle changes in the growth process of vegetation affected by pests and diseases, resulting in poor accuracy and reliability of vegetation health detection. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the quality of the collected images is often low, especially when the vegetation is affected by pests and diseases, it is difficult to timely detect subtle changes in the vegetation growth process caused by pests and diseases, resulting in poor accuracy and reliability of vegetation health detection, and to propose a vegetation growth status monitoring and identification method and device based on image vision.

[0005] In a first aspect of the present invention, a vegetation growth status monitoring and identification method based on image vision is first proposed, the method comprising:

[0006] Acquire an initial RGB image and an initial thermal infrared image of the target vegetation, and preprocess the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively;

[0007] Identify a target interval in the target RGB image, and cut the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image; the target interval is an image area where leaves are located in the target RGB image;

[0008] Performing fusion feature extraction on the first image and the second image to obtain a first feature image, and performing feature enhancement on the first feature image to obtain a second feature image;

[0009] The second feature image is converted into a target image through a decoder, and feature extraction is performed on the target image to obtain target features. Status information of corresponding vegetation in a preset database is searched according to the target features, and an evaluation result is determined according to the status information.

[0010] Optionally, identifying a target interval in the target RGB image includes:

[0011] For each pixel in the target RGB image, an index parameter corresponding to the pixel is obtained according to the RGB value corresponding to the pixel, a target index value is classified for the pixel according to the index parameter, and a target index image is obtained according to the classification results of all pixels;

[0012] Obtain a target index mean value according to all target index values ​​in the target index image, and obtain a target threshold value according to the maximum target index value in the target index image and the target index mean value;

[0013] For each pixel in the target index image, if the target index value of the pixel is greater than or equal to the target threshold, the pixel is marked as a vegetation pixel; if the target index value of the pixel is less than the target threshold, the pixel is marked as a background pixel;

[0014] All vegetation pixels are obtained to form the target interval.

[0015] Optionally, performing fusion feature extraction on the first image and the second image to obtain a first feature image includes:

[0016] Step 1: splicing the first image and the second image in the channel dimension to obtain a first spliced ​​image set, and determining a first activation image according to the first spliced ​​image set; the number of channel images in the channel image set is determined by the image with the most channels;

[0017] Step 2: stitching the first image and the first activation image in the channel dimension to obtain a second stitched image set, and determining a first ready-to-activate image according to the second stitched image set;

[0018] Step 3: stitching the second image and the first activation image in the channel dimension to obtain a third stitched image set, and determining a second ready-to-activate image according to the third stitched image set;

[0019] Step 4: fusing the first activation preparation image and the second activation preparation image to obtain a second activation image;

[0020] Step 5: record the second activated image as the first activated image, and repeat the above steps 2, 3 and 4 until the preset conditions are met to obtain the first feature image.

[0021] Optionally, the A image and the B image are spliced ​​in the channel dimension to obtain a C image set; when the A image is the first image and the B image is the second image, the C image set is the first spliced ​​image set; when the A image is the first image and the B image is the first activated image, the C image set is the second spliced ​​image set; when the A image is the second image and the B image is the first activated image, the C image set is the third spliced ​​image set; when the A image is the first ready-to-activate image and the B image is the second ready-to-activate image, the C image set is the second activated image set; comprising:

[0022] Performing channel alignment on the A image and the B image to obtain a channel image set;

[0023] For each channel image in the channel image set, multi-core parallel convolution is performed on the channel image to obtain an initial convolution image set; all initial convolution images in the initial convolution image set are normalized to obtain a convolution image set;

[0024] For each convolution image in the convolution image set, the convolution image is activated by an activation function and then a pooling operation is performed to obtain a pooling map;

[0025] Calculate the L2 norm of each pooling map, sort all pooling maps from high to low according to the L2 norm of each pooling map to obtain a pooling map sequence set, obtain a valid pooling map in the pooling map sequence set according to a preset condition, and perform weighted summation on all valid pooling maps to obtain a C image;

[0026] Get the C images corresponding to all channel images to get the C image set.

[0027] Optionally, performing feature enhancement on the first feature image to obtain a second feature image includes:

[0028] Substituting the first feature image into a 3×3 convolutional layer to obtain three identical feature maps, which are recorded as a first identical feature map, a second identical feature map, and a third identical feature map;

[0029] Obtain a first enhanced feature map by multiplying the maximum eigenvalue by the first identical feature map;

[0030] The second same feature map and the third same feature map are processed by a Sigmoid function to obtain a second enhanced feature map and a third enhanced feature map;

[0031] The first enhanced feature map, the second enhanced feature map and the third enhanced feature map are concatenated to obtain an initial enhanced feature map, and the initial enhanced feature map is substituted into a 1×1 convolutional layer to obtain a second feature image.

[0032] In a second aspect of the present invention, a vegetation growth status monitoring and identification device based on image vision is proposed, comprising:

[0033] An image preprocessing module is used to obtain an initial RGB image and an initial thermal infrared image of the target vegetation, and preprocess the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively;

[0034] An image cutting module is used to identify a target interval in the target RGB image, and cut the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image; the target interval is an image area where leaves are located in the target RGB image;

[0035] A feature enhancement module, configured to extract fusion features of the first image and the second image to obtain a first feature image, and to enhance features of the first feature image to obtain a second feature image;

[0036] The status information determination module is used to convert the second feature image into a target image through a decoder, extract features from the target image to obtain target features, search for status information of corresponding vegetation in a preset database according to the target features, and determine an evaluation result according to the status information.

[0037] Optionally, the image cutting module includes:

[0038] A target index image generation module is used to obtain an index parameter corresponding to each pixel in the target RGB image according to the RGB value corresponding to the pixel, classify the pixel into a target index value according to the index parameter, and obtain a target index image according to the classification results of all pixels;

[0039] A target threshold generating module, used to obtain a target index mean value according to all target index values ​​in the target index image, and to obtain a target threshold value according to a maximum target index value in the target index image and the target index mean value;

[0040] a pixel classification module, for each pixel in the target index image, marking the pixel as a vegetation pixel if the target index value of the pixel is greater than or equal to the target threshold, and marking the pixel as a background pixel if the target index value of the pixel is less than the target threshold;

[0041] The target interval generation module is used to obtain all vegetation pixels to form a target interval.

[0042] Optionally, the feature enhancement module includes:

[0043] A first activation image generation module is used to splice the first image and the second image in the channel dimension to obtain a first spliced ​​image set, and determine a first activation image according to the first spliced ​​image set; the number of channel images in the channel image set is determined by the image with the most channels;

[0044] a first ready-to-activate image generating module, configured to stitch the first image and the first activated image in a channel dimension to obtain a second stitched image set, and determine a first ready-to-activate image according to the second stitched image set;

[0045] a second ready-to-activate image generating module, configured to stitch the second image and the first activated image in a channel dimension to obtain a third stitched image set, and determine a second ready-to-activate image according to the third stitched image set;

[0046] A second activation image generating module, configured to fuse the first activation preparation image and the second activation preparation image to obtain a second activation image;

[0047] The loop termination module is used to record the second activation image as the first activation image, and return to the first preparation activation image generation module, the second preparation activation image generation module and the second activation image generation module until the preset conditions are met to obtain the first feature image.

[0048] Optionally, the target module includes: when the target module is the first activation image generation module, the A image is the first image, the B image is the second image, and the C image set is the first spliced ​​image set; when the target module is the first preparation activation image generation module, the A image is the first image, the B image is the first activation image, and the C image is the second spliced ​​image set; when the target module is the second preparation activation image generation module, the A image is the second image, the B image is the first activation image, and the C image is the third spliced ​​image set; when the target module is the second activation image generation module, the A image is the first preparation activation image, the B image is the second preparation activation image, and the C image is the second activation image set;

[0049] A channel alignment module is used to perform channel alignment on the A image and the B image to obtain a channel image set;

[0050] A channel image processing module is used for performing multi-core parallel convolution on each channel image in the channel image set to obtain an initial convolution image set; and performing normalization processing on all initial convolution images in the initial convolution image set to obtain a convolution image set;

[0051] A convolution image processing module, used for activating each convolution image in the convolution image set through an activation function and then performing a pooling operation on the convolution image to obtain a pooling image;

[0052] A C image generation module is used to calculate the L2 norm of each pooling map, sort all pooling maps from high to low according to the L2 norm of each pooling map to obtain a pooling map sequence set, obtain a valid pooling map in the pooling map sequence set according to a preset condition, and perform weighted summation on all valid pooling maps to obtain a C image;

[0053] The C image set determination module is used to obtain the C images corresponding to all channel images to obtain the C image set.

[0054] Optionally, the feature enhancement module further includes:

[0055] A same feature map generation module, used for substituting the first feature image into a 3×3 convolutional layer to obtain three same feature maps, which are recorded as a first same feature map, a second same feature map and a third same feature map;

[0056] A first enhancement module, configured to obtain a first enhanced feature map by multiplying the maximum eigenvalue by the first identical feature map;

[0057] A second enhancement module, used for processing the second same feature map and the third same feature map respectively through a Sigmoid function to obtain a second enhanced feature map and a third enhanced feature map;

[0058] The second feature image generation module is used to splice the first enhanced feature map, the second enhanced feature map and the third enhanced feature map to obtain an initial enhanced feature map, and substitute the initial enhanced feature map into a 1×1 convolution layer to obtain a second feature image.

[0059] Beneficial effects of the present invention:

[0060] The present invention proposes a vegetation growth status monitoring and identification method based on image vision. The method comprises the following steps: obtaining an initial RGB image and an initial thermal infrared image of a target vegetation, preprocessing the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively; identifying a target interval in the target RGB image, and cutting the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image; fusing the first image and the second image to extract features to obtain a first feature image, and enhancing the features of the first feature image to obtain a second feature image; converting the second feature image into a target image through a decoder, extracting features of the target image to obtain target features, searching for status information of corresponding vegetation in a preset database according to the target features, and determining an evaluation result according to the status information. The target RGB image and the target thermal infrared image are cut by the target interval, which retains the area where the leaves are located, improves the image quality, reduces noise interference, and helps to improve the efficiency and accuracy of subsequent image processing. The RGB image and the thermal infrared image are combined for feature extraction, and the complementary information of the two images is fully utilized to improve the accuracy of vegetation health detection. Feature extraction is then performed on the image after feature fusion to retain the detailed features to the greatest extent. The status information of the vegetation in the preset database is searched according to the target features, which further improves the accuracy of vegetation health detection and improves the accuracy and reliability of vegetation health detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present invention will be further described below in conjunction with the accompanying drawings.

[0062] Figure 1 A flowchart of a vegetation growth status monitoring and identification method based on image vision is provided for an embodiment of the present invention;

[0063] Figure 2 A structural schematic diagram of a vegetation growth status monitoring and identification device based on image vision is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0064] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0065] The embodiment of the present invention provides a method for monitoring and identifying vegetation growth status based on image vision. Figure 1 , Figure 1 A flowchart of a vegetation growth status monitoring and identification method based on image vision provided by an embodiment of the present invention. The method comprises the following steps:

[0066] S101, acquiring an initial RGB image and an initial thermal infrared image of a target vegetation, and preprocessing the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively;

[0067] S102, identifying a target interval in the target RGB image, and cutting the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image;

[0068] S103, performing fusion feature extraction on the first image and the second image to obtain a first feature image, and performing feature enhancement on the first feature image to obtain a second feature image;

[0069] S104, converting the second feature image into a target image through a decoder, extracting features from the target image to obtain target features, searching for status information of corresponding vegetation in a preset database according to the target features, and determining an evaluation result according to the status information.

[0070] The target interval is the image area where the leaves are located in the target RGB image.

[0071] A vegetation growth status monitoring and identification method based on image vision is provided based on an embodiment of the present invention. The target RGB image and the target thermal infrared image are cut by the target interval, so that the area where the leaves are located is retained, the image quality is improved, and the noise interference is reduced, which is helpful to improve the efficiency and accuracy of subsequent image processing. The RGB image and the thermal infrared image are combined for feature extraction, and the complementary information of the two images is fully utilized to improve the accuracy of vegetation health status detection. Then, feature extraction is performed on the image after feature fusion, and the detail features are retained to the greatest extent. The status information of the vegetation in the preset database is searched according to the target features, which further improves the accuracy of vegetation health status detection and improves the accuracy and reliability of vegetation health detection.

[0072] In one implementation, RGB images and thermal infrared images are combined for feature extraction, making full use of the complementary information of the two images to improve the accuracy of vegetation health detection. The extracted features are enhanced to highlight the texture details and target information of the vegetation, further improving the precision and reliability of the analysis.

[0073] In one implementation, the target vegetation is the vegetation to be photographed, and an initial RGB image and an initial thermal infrared image of the target vegetation are obtained. The size of the two images is WxH pixels, and the direction is consistent. The two images capture the same content, and the corresponding pixel positions correspond one to one, that is, the pixels at the same position represent the content of the same physical position in the two images.

[0074] In one implementation method, for RGB images, contrast stretching, histogram equalization and other methods are used to enhance the visual effect of RGB images, making the vegetation area clearer, and Gaussian filtering is used to denoise the RGB images; for thermal infrared images, temperature information is converted into a visual color image, and then a smoothing filter is used to remove noise in the thermal infrared images.

[0075] In one implementation, the decoder includes a deconvolution layer, a convolution layer, an activation function and normalization. The deconvolution layer is used to gradually increase the spatial resolution of the feature map. The convolution layer is used to further process the feature map to make it closer to the structure of the original image. The activation function usually uses ReLU or Leaky ReLU activation function to ensure nonlinearity. Normalization is used to standardize the feature map to facilitate training.

[0076] In one implementation, feature extraction of a target image to obtain a target feature is to substitute the target image into a CNN convolutional neural network model for feature extraction to obtain the target feature. The CNN convolutional neural network model includes an initial convolution layer, a maximum pooling layer, a residual block stage, a global average pooling layer, and a fully connected layer. The initial convolution layer is a 7×7 convolution kernel with a stride of 2 and an output channel number of 64. The maximum pooling layer is a 3×3 pooling kernel with a stride of 2. The residual block stage includes a first stage: 1 residual block including 2 3×3 convolution layers, a second stage: 3 residual blocks, each residual block including 2 3×3 convolution layers, a third stage: 4 residual blocks, each residual block including 2 3×3 convolution layers, and a fourth stage: 6 residual blocks, each residual block including 2 3×3 convolution layers.

[0077] In one implementation, the preset database is the target features obtained after training the historical image data through the above steps, and the plant health status corresponding to the historical image data is saved together with the target features corresponding to the historical image data in the preset database.

[0078] In one implementation, preprocessing and target interval cutting improve image quality and reduce noise interference, which helps to improve the efficiency and accuracy of subsequent image processing. Database comparison and status information search can provide a basis for evaluation results and decision-making, helping to carry out effective vegetation management and maintenance.

[0079] In one embodiment, identifying a target interval in a target RGB image includes:

[0080] For each pixel in the target RGB image, an index parameter corresponding to the pixel is obtained according to the RGB value corresponding to the pixel, and a target index value is classified for the pixel according to the index parameter, and a target index image is obtained according to the classification results of all pixels;

[0081] A target index mean is obtained according to all target index values ​​in the target index image, and a target threshold is obtained according to the maximum target index value and the target index mean value in the target index image;

[0082] For each pixel in the target index image, if the target index value of the pixel is greater than or equal to the target threshold, the pixel is marked as a vegetation pixel; if the target index value of the pixel is less than the target threshold, the pixel is marked as a background pixel;

[0083] All vegetation pixels are obtained to form the target interval.

[0084] In one implementation, by analyzing the RGB value of each pixel, the target index parameter of each pixel can be accurately calculated, thereby accurately locating the vegetation area and avoiding misjudgment and missed judgment.

[0085] In one implementation, the index parameter is 2G-BR, G is the green channel value corresponding to the pixel, B is the blue channel value corresponding to the pixel, R is the red channel value corresponding to the pixel, and the target index value of the pixel is classified as follows: if 2G-BR>=255, the corresponding value of the pixel is recorded as 255; if 0<2G-BR<255, the corresponding value of the pixel is recorded as the value corresponding to the index parameter; if 2G-BR<=0, the corresponding value of the pixel is recorded as 0.

[0086] In one implementation, the formula Get the target threshold; is the target threshold, is the target index mean, is the maximum target index value. When there is a pixel whose index parameter is greater than or equal to 255, the maximum target index value is 255. is a constant in the range of [-0.5, 0.5], and the specific value is determined by the technician.

[0087] In one implementation, the target threshold is dynamically calculated using the target index mean and the maximum target index value, so that the method has strong adaptability and can be adjusted according to the characteristics of different images, thereby improving robustness.

[0088] In one implementation, if there are background pixels in the area formed by the vegetation pixels and the number of background pixels is less than a preset number, the background pixel area is recorded as the vegetation pixel area. If the number of background pixels is greater than or equal to the preset number, the target threshold is adjusted and each pixel in the target index image is re-judged. The preset number is determined by the technician.

[0089] In one implementation, an index parameter classification and threshold segmentation method is used to ensure accurate distinction between vegetation and background, improve recognition accuracy, and effectively reduce noise interference.

[0090] In one embodiment, fusing the first image and the second image to extract features to obtain the first feature image includes:

[0091] Step 1: splicing the first image and the second image in the channel dimension to obtain a first spliced ​​image set, and determining a first activation image according to the first spliced ​​image set; the number of channel images in the channel image set is determined by the image with the most channels;

[0092] Step 2: splicing the first image and the first activation image in the channel dimension to obtain a second spliced ​​image set, and determining the first ready-to-activate image according to the second spliced ​​image set;

[0093] Step 3: splicing the second image and the first activation image in the channel dimension to obtain a third spliced ​​image set, and determining the second ready-to-activate image according to the third spliced ​​image set;

[0094] Step 4: Fusing the first activation preparation image and the second activation preparation image to obtain a second activation image;

[0095] Step 5: record the second activated image as the first activated image, and repeat the above steps 2, 3 and 4 until the preset conditions are met to obtain the first feature image.

[0096] In one implementation, by splicing the RGB image and the thermal infrared image in the channel dimension, the complementary information of the two different types of images is fully utilized, thereby improving the comprehensiveness and accuracy of feature extraction.

[0097] In one implementation, the activated image is determined based on the image set, the L2 norm of each image is calculated, all images are sorted from high to low according to the L2 norm, and the first 20 percent of images are obtained for weighted summation to obtain the activated image.

[0098] In one implementation, a weighted sum is performed on the two images to obtain a fused image, wherein the weight of the first image to be activated is A, the weight of the second image to be activated is B, A+B=1, and the values ​​of A and B are determined by a technician.

[0099] In one implementation, multiple stitching and activation iterations are used to continuously optimize and enhance the feature extraction results, effectively align and fuse the feature information of different images, improve the image fusion effect, and make the final feature image clearer and more consistent.

[0100] In one embodiment, image A and image B are spliced ​​in the channel dimension to obtain a C image set; when image A is the first image and image B is the second image, image C is the first spliced ​​image set; when image A is the first image and image B is the first activated image, image C is the second spliced ​​image set; when image A is the second image and image B is the first activated image, image C is the third spliced ​​image set; when image A is the first ready-to-activate image and image B is the second ready-to-activate image, image C is the second activated image set; including:

[0101] Perform channel alignment on image A and image B to obtain a channel image set;

[0102] For each channel image in the channel image set, multi-core parallel convolution is performed on the channel image to obtain an initial convolution image set; all initial convolution images in the initial convolution image set are normalized to obtain a convolution image set;

[0103] For each convolution image in the convolution image set, the convolution image is activated by an activation function and then a pooling operation is performed to obtain a pooling map;

[0104] Calculate the L2 norm of each pooling map, sort all pooling maps from high to low according to the L2 norm of each pooling map to obtain a pooling map sequence set, obtain the valid pooling map in the pooling map sequence set according to the preset conditions, and perform weighted summation on all valid pooling maps to obtain the C image;

[0105] Get the C images corresponding to all channel images to get the C image set.

[0106] In one implementation, the initial convolution image is normalized to ensure that the eigenvalues ​​between different channels and images are on the same scale, avoiding the deviation caused by different eigenvalue ranges and improving the accuracy of feature extraction.

[0107] In one implementation, the preset condition is determined by a technician and may be the first 30 percent of images in the arrangement.

[0108] In one implementation, the noise information in the image is effectively filtered out through activation functions and pooling operations, the signal-to-noise ratio of the feature map is improved, and the reliability of feature extraction is enhanced.

[0109] In one implementation, the L2 norm is used to sort and select the pooling map, which ensures that the extracted feature map contains more useful information, eliminates irrelevant or noise features, and improves the representativeness and effectiveness of the feature map.

[0110] In one embodiment, performing feature enhancement on the first feature image to obtain the second feature image includes:

[0111] Substitute the first feature image into the 3×3 convolutional layer to obtain three identical feature maps, which are recorded as the first identical feature map, the second identical feature map, and the third identical feature map;

[0112] Obtain a first enhanced feature map by multiplying the maximum eigenvalue by the first identical feature map;

[0113] The second same feature map and the third same feature map are processed by the Sigmoid function to obtain a second enhanced feature map and a third enhanced feature map;

[0114] The first enhanced feature map, the second enhanced feature map and the third enhanced feature map are concatenated to obtain an initial enhanced feature map, and the initial enhanced feature map is substituted into a 1×1 convolutional layer to obtain a second feature image.

[0115] In one implementation, the first feature image is divided into three identical feature images, and different feature details are enhanced through different processing methods, so that the final feature image can better retain and highlight information.

[0116] In one implementation, the RGB image and the thermal infrared image are extracted through a convolution layer to generate a response value in a feature map, all response values ​​of the RGB image and the thermal infrared image are compared, and the maximum response value is recorded as the maximum feature value.

[0117] In one implementation, the RGB image and the thermal infrared image are processed respectively by the maximum eigenvalue and the Sigmoid function, which fully utilizes the features of the two image sources and improves the diversity and accuracy of feature extraction.

[0118] In one implementation, the enhanced feature maps obtained by three different processing methods are spliced ​​together to form an initial enhanced feature map, which effectively combines the advantages of different feature enhancement methods so that the final feature image contains richer information.

[0119] In one implementation, the number of channels of the initial enhanced feature map is compressed through a 1×1 convolutional layer, which reduces the complexity of the feature map while retaining key feature information, improving computational efficiency and simplicity of subsequent processing.

[0120] Based on the same inventive concept, the embodiment of the present invention also provides a vegetation growth status monitoring and identification device based on image vision. Figure 2 , Figure 2 A schematic diagram of a vegetation growth status monitoring and identification device based on image vision provided by an embodiment of the present invention includes:

[0121] An image preprocessing module is used to obtain an initial RGB image and an initial thermal infrared image of the target vegetation, and preprocess the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively;

[0122] An image cutting module is used to identify a target interval in a target RGB image, and cut the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image; the target interval is an image area where leaves are located in the target RGB image;

[0123] A feature enhancement module, used for fusing the first image and the second image to extract features to obtain a first feature image, and performing feature enhancement on the first feature image to obtain a second feature image;

[0124] The state information determination module is used to convert the second feature image into a target image through a decoder, extract features from the target image to obtain target features, search for state information of corresponding vegetation in a preset database according to the target features, and determine an evaluation result according to the state information.

[0125] A vegetation growth status monitoring and identification device based on image vision provided by an embodiment of the present invention cuts a target RGB image and a target thermal infrared image through a target interval, retains the area where the leaves are located, improves image quality, reduces noise interference, and helps to improve the efficiency and accuracy of subsequent image processing. Feature extraction is performed on the RGB image and the thermal infrared image, and the complementary information of the two images is fully utilized to improve the accuracy of vegetation health status detection. Feature extraction is then performed on the image after feature fusion to retain the detail features to the greatest extent. The status information of the vegetation in the preset database is searched according to the target features, which further improves the accuracy of vegetation health status detection and improves the accuracy and reliability of vegetation health detection.

[0126] In one embodiment, the image cutting module includes:

[0127] A target index image generation module is used to obtain an index parameter corresponding to each pixel in the target RGB image according to the RGB value corresponding to the pixel, classify the pixel into a target index value according to the index parameter, and obtain a target index image according to the classification results of all pixels;

[0128] A target threshold generation module is used to obtain a target index mean value according to all target index values ​​in the target index image, and to obtain a target threshold value according to a maximum target index value and a target index mean value in the target index image;

[0129] A pixel classification module is used for marking each pixel in the target index image as a vegetation pixel if the target index value of the pixel is greater than or equal to the target threshold, and marking the pixel as a background pixel if the target index value of the pixel is less than the target threshold;

[0130] The target interval generation module is used to obtain all vegetation pixels to form a target interval.

[0131] In one embodiment, the feature enhancement module includes:

[0132] A first activation image generation module is used to splice the first image and the second image in the channel dimension to obtain a first spliced ​​image set, and determine the first activation image according to the first spliced ​​image set; the number of channel images in the channel image set is determined by the image with the most channels;

[0133] A first ready-to-activate image generating module, configured to stitch the first image and the first activated image in a channel dimension to obtain a second stitched image set, and determine a first ready-to-activate image according to the second stitched image set;

[0134] a second ready-to-activate image generating module, configured to splice the second image and the first activated image in a channel dimension to obtain a third spliced ​​image set, and determine a second ready-to-activate image according to the third spliced ​​image set;

[0135] A second activation image generating module, configured to fuse the first activation preparation image and the second activation preparation image to obtain a second activation image;

[0136] The loop termination module is used to record the second activation image as the first activation image, and return to the first preparation activation image generation module, the second preparation activation image generation module and the second activation image generation module until the preset conditions are met to obtain the first feature image.

[0137] In one embodiment, the target module includes: when the target module is a first activation image generation module, image A is the first image, image B is the second image, and image C is the first spliced ​​image set; when the target module is a first ready-to-activate image generation module, image A is the first image, image B is the first activation image, and image C is the second spliced ​​image set; when the target module is a second ready-to-activate image generation module, image A is the second image, image B is the first activation image, and image C is the third spliced ​​image set; when the target module is a second activation image generation module, image A is the first ready-to-activate image, image B is the second ready-to-activate image, and image C is the second activation image set;

[0138] A channel alignment module is used to perform channel alignment on the A image and the B image to obtain a channel image set;

[0139] The channel image processing module is used for performing multi-core parallel convolution on each channel image in the channel image set to obtain an initial convolution image set; and performing normalization processing on all initial convolution images in the initial convolution image set to obtain a convolution image set;

[0140] A convolution image processing module, for activating each convolution image in the convolution image set through an activation function and then performing a pooling operation on the convolution image to obtain a pooling map;

[0141] A C image generation module is used to calculate the L2 norm of each pooling map, sort all pooling maps from high to low according to the L2 norm of each pooling map to obtain a pooling map sequence set, obtain valid pooling maps in the pooling map sequence set according to preset conditions, and perform weighted summation on all valid pooling maps to obtain a C image;

[0142] The C image set determination module is used to obtain the C images corresponding to all channel images to obtain the C image set.

[0143] In one embodiment, the feature enhancement module further includes:

[0144] A same feature map generation module is used to substitute the first feature image into the 3×3 convolution layer to obtain three same feature maps, which are recorded as a first same feature map, a second same feature map, and a third same feature map;

[0145] A first enhancement module, configured to obtain a first enhanced feature map by multiplying the maximum eigenvalue by the first identical feature map;

[0146] A second enhancement module is used to process the second same feature map and the third same feature map respectively through a Sigmoid function to obtain a second enhanced feature map and a third enhanced feature map;

[0147] The second feature image generation module is used to splice the first enhanced feature map, the second enhanced feature map and the third enhanced feature map to obtain an initial enhanced feature map, and substitute the initial enhanced feature map into the 1×1 convolution layer to obtain the second feature image.

[0148] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A vegetation growth status monitoring and identification method based on image vision, characterized in that: The method comprises: Acquire an initial RGB image and an initial thermal infrared image of the target vegetation, and preprocess the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively; Identify a target interval in the target RGB image, and cut the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image; the target interval is an image area where leaves are located in the target RGB image; Performing fusion feature extraction on the first image and the second image to obtain a first feature image, and performing feature enhancement on the first feature image to obtain a second feature image; The second feature image is converted into a target image through a decoder, a feature of the target image is extracted to obtain a target feature, state information of corresponding vegetation in a preset database is searched according to the target feature, and an evaluation result is determined according to the state information; Identifying a target interval in the target RGB image includes: For each pixel in the target RGB image, an index parameter corresponding to the pixel is obtained according to the RGB value corresponding to the pixel, a target index value is classified for the pixel according to the index parameter, and a target index image is obtained according to the classification results of all pixels; Obtain a target index mean value according to all target index values ​​in the target index image, and obtain a target threshold value according to the maximum target index value in the target index image and the target index mean value; For each pixel in the target index image, if the target index value of the pixel is greater than or equal to the target threshold, the pixel is marked as a vegetation pixel; if the target index value of the pixel is less than the target threshold, the pixel is marked as a background pixel; All vegetation pixels are obtained to form the target interval.

2. According to the method for monitoring and identifying vegetation growth status based on image vision in claim 1, it is characterized in that: The step of fusing the first image and the second image to extract a first feature image includes: Step 1: stitching the first image and the second image in the channel dimension to obtain a first stitched image set, and determining a first activation image according to the first stitched image set; Step 2: stitching the first image and the first activation image in the channel dimension to obtain a second stitched image set, and determining a first ready-to-activate image according to the second stitched image set; Step 3: stitching the second image and the first activation image in the channel dimension to obtain a third stitched image set, and determining a second ready-to-activate image according to the third stitched image set; Step 4: fusing the first activation preparation image and the second activation preparation image to obtain a second activation image; Step 5: record the second activation image as the first activation image, and repeat the above steps 2, 3 and 4 until the preset conditions are met to obtain the first feature image.

3. The vegetation growth status monitoring and identification method based on image vision according to claim 2 is characterized in that: The A image and the B image are spliced ​​in the channel dimension to obtain a C image set; when the A image is the first image and the B image is the second image, the C image set is the first spliced ​​image set; when the A image is the first image and the B image is the first activated image, the C image set is the second spliced ​​image set; when the A image is the second image and the B image is the first activated image, the C image set is the third spliced ​​image set; When the A image is a first ready-to-activate image and the B image is a second ready-to-activate image, the C image set is a second activated image set; comprising: Perform channel alignment on the A image and the B image to obtain a channel image set; the number of channel images in the channel image set is determined by the image with the most channels; For each channel image in the channel image set, multi-core parallel convolution is performed on the channel image to obtain an initial convolution image set; all initial convolution images in the initial convolution image set are normalized to obtain a convolution image set; For each convolution image in the convolution image set, the convolution image is activated by an activation function and then a pooling operation is performed to obtain a pooling map; Calculate the L2 norm of each pooling map, sort all pooling maps from high to low according to the L2 norm of each pooling map to obtain a pooling map sequence set, obtain a valid pooling map in the pooling map sequence set according to a preset condition, and perform weighted summation on all valid pooling maps to obtain a C image; Get the C images corresponding to all channel images to get the C image set.

4. The vegetation growth status monitoring and identification method based on image vision according to claim 1 is characterized in that: The first feature image is subjected to feature enhancement to obtain a second feature image, comprising: Substituting the first feature image into a 3×3 convolutional layer to obtain three identical feature maps, which are recorded as a first identical feature map, a second identical feature map, and a third identical feature map; Obtain a first enhanced feature map by multiplying the maximum eigenvalue by the first identical feature map; The second same feature map and the third same feature map are processed by a Sigmoid function to obtain a second enhanced feature map and a third enhanced feature map; The first enhanced feature map, the second enhanced feature map and the third enhanced feature map are concatenated to obtain an initial enhanced feature map, and the initial enhanced feature map is substituted into a 1×1 convolutional layer to obtain a second feature image.

5. A vegetation growth status monitoring and identification device based on image vision, characterized in that: The device comprises: An image preprocessing module is used to obtain an initial RGB image and an initial thermal infrared image of the target vegetation, and preprocess the initial RGB image and the initial thermal infrared image to obtain a target RGB image and a target thermal infrared image respectively; An image cutting module is used to identify a target interval in the target RGB image, and cut the target RGB image and the target thermal infrared image according to the target interval to obtain a first image and a second image; the target interval is an image area where leaves are located in the target RGB image; A feature enhancement module, configured to extract fusion features of the first image and the second image to obtain a first feature image, and to enhance features of the first feature image to obtain a second feature image; a state information determination module, configured to convert the second feature image into a target image through a decoder, extract features from the target image to obtain target features, search for state information of corresponding vegetation in a preset database according to the target features, and determine an evaluation result according to the state information; The image cutting module comprises: A target index image generation module is used to obtain an index parameter corresponding to each pixel in the target RGB image according to the RGB value corresponding to the pixel, classify the pixel into a target index value according to the index parameter, and obtain a target index image according to the classification results of all pixels; A target threshold generating module, used to obtain a target index mean value according to all target index values ​​in the target index image, and to obtain a target threshold value according to a maximum target index value in the target index image and the target index mean value; a pixel classification module, for each pixel in the target index image, marking the pixel as a vegetation pixel if the target index value of the pixel is greater than or equal to the target threshold, and marking the pixel as a background pixel if the target index value of the pixel is less than the target threshold; The target interval generation module is used to obtain all vegetation pixels to form a target interval.

6. The vegetation growth status monitoring and identification device based on image vision according to claim 5 is characterized in that: The feature enhancement module comprises: A first activation image generation module, configured to stitch the first image and the second image in a channel dimension to obtain a first stitched image set, and determine a first activation image according to the first stitched image set; a first ready-to-activate image generating module, configured to stitch the first image and the first activated image in a channel dimension to obtain a second stitched image set, and determine a first ready-to-activate image according to the second stitched image set; a second ready-to-activate image generating module, configured to stitch the second image and the first activated image in a channel dimension to obtain a third stitched image set, and determine a second ready-to-activate image according to the third stitched image set; A second activation image generating module, configured to fuse the first activation preparation image and the second activation preparation image to obtain a second activation image; The loop termination module is used to record the second activation image as the first activation image, and return to the first preparation activation image generation module, the second preparation activation image generation module and the second activation image generation module until the preset conditions are met to obtain the first feature image.

7. The vegetation growth status monitoring and identification device based on image vision according to claim 6 is characterized in that: The target module includes: when the target module is the first activation image generation module, the A image is the first image, the B image is the second image, and the C image set is the first spliced ​​image set; when the target module is the first preparation activation image generation module, the A image is the first image, the B image is the first activation image, and the C image is the second spliced ​​image set; when the target module is the second preparation activation image generation module, the A image is the second image, the B image is the first activation image, and the C image is the third spliced ​​image set; when the target module is the second activation image generation module, the A image is the first preparation activation image, the B image is the second preparation activation image, and the C image is the second activation image set; A channel alignment module is used to perform channel alignment on the A image and the B image to obtain a channel image set; the number of channel images in the channel image set is determined by the image with the most channels; A channel image processing module is used for performing multi-core parallel convolution on each channel image in the channel image set to obtain an initial convolution image set; and performing normalization processing on all initial convolution images in the initial convolution image set to obtain a convolution image set; A convolution image processing module, used for activating each convolution image in the convolution image set through an activation function and then performing a pooling operation on the convolution image to obtain a pooling image; A C image generation module is used to calculate the L2 norm of each pooling map, sort all pooling maps from high to low according to the L2 norm of each pooling map to obtain a pooling map sequence set, obtain a valid pooling map in the pooling map sequence set according to a preset condition, and perform weighted summation on all valid pooling maps to obtain a C image; The C image set determination module is used to obtain the C images corresponding to all channel images to obtain the C image set.

8. The vegetation growth status monitoring and identification device based on image vision according to claim 5 is characterized in that: The feature enhancement module also includes: A same feature map generation module, used for substituting the first feature image into a 3×3 convolutional layer to obtain three same feature maps, which are recorded as a first same feature map, a second same feature map and a third same feature map; A first enhancement module, configured to obtain a first enhanced feature map by multiplying the maximum eigenvalue by the first identical feature map; A second enhancement module, used for processing the second same feature map and the third same feature map respectively through a Sigmoid function to obtain a second enhanced feature map and a third enhanced feature map; The second feature image generation module is used to splice the first enhanced feature map, the second enhanced feature map and the third enhanced feature map to obtain an initial enhanced feature map, and substitute the initial enhanced feature map into a 1×1 convolution layer to obtain a second feature image.

Images