Night plant image processing method, device and equipment
Through the nocturnal plant image processing method, the combination technology of deformation nucleus and dynamic modulation modules is used to solve the problem of low image segmentation accuracy at nocturnal plant, and the accuracy of plant morphological phenotype parameters extraction and image recognition are achieved, which is suitable for complex High-throughput monitoring of agricultural scenarios.
Patent Information
- Application Number
- CN202411040144.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to accurately extract plant phenotypes under night conditions, especially in complex field environments. Image segmentation methods are difficult to process plant images of different scales, postures, perspectives and deformations, resulting in insufficient accuracy of segmentation accuracy and parameter extraction. .
A method of image processing at nighttime plants is proposed. The first feature map is obtained through encoding processing, and the dynamic features in the form of deformation nucleus is used for upsampling. The amplitude of the feature position is adjusted in combination with the dynamic modulation module to generate an upsampling feature map, and then determine Plant morphological phenotypic parameter data.
The accuracy and graph recognition of plant morphological phenotype parameters are improved, making the analysis based on plant morphological phenotype parameters more accurate, and is suitable for high-throughput plant phenotype monitoring in complex agricultural scenarios.
Smart Images

Figure CN118918478A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, device and equipment for processing nighttime plant images. Background Art
[0002] In modern agriculture, plant phenotyping is an important means to study plant growth, development and ability to adapt to the environment.
[0003] However, the phenotypic extraction of many field plants is greatly affected by light, and the accuracy is difficult to meet practical applications. Under relatively stable nighttime light conditions, the controllability of phenotypic extraction is higher, but the existing daytime phenotypic extraction methods are difficult to apply to nighttime phenotypic extraction. Summary of the invention
[0004] In view of this, the purpose of this application is to propose a nighttime plant image processing method, device and equipment to solve or partially solve the above-mentioned technical problems.
[0005] Based on the above purpose, the present application provides a method for processing nighttime plant images, comprising:
[0006] Encoding the nighttime plant image to obtain a first feature map;
[0007] After performing an offset process based on the first feature map, upsampling is performed using dynamic features in the form of a deformation kernel to obtain a second feature map;
[0008] Using a dynamic modulation module to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map;
[0009] An up-sampling mask image corresponding to the up-sampling feature image is determined, plant phenotype extraction is performed on the up-sampling mask image to obtain plant morphological phenotype parameter data, and the plant morphological phenotype parameter data is output.
[0010] Based on the same inventive concept, the present application also provides a nighttime plant image processing device, comprising:
[0011] An encoding module is configured to encode the nighttime plant image to obtain a first feature map;
[0012] an offset processing module, configured to perform an offset process based on the first feature map, and then perform an upsampling process using a dynamic feature in the form of a deformation kernel to obtain a second feature map;
[0013] A dynamic modulation module is configured to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map;
[0014] The plant phenotype extraction module is configured to determine an up-sampled mask image corresponding to the up-sampled feature image, perform plant phenotype extraction on the up-sampled mask image, obtain plant morphological phenotype parameter data, and output the plant morphological phenotype parameter data.
[0015] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.
[0016] From the above, it can be seen that the night plant image processing method, device and equipment provided by the present application can encode night plant images and unify the night plant images into a first feature map that can be subsequently processed; then the first feature map is first offset processed, and then up-sampled using dynamic features in the form of a deformation kernel to obtain a second feature map that is more in line with the plant morphology; in order to further improve the recognition of the second feature map, the dynamic modulation module is used to adjust the amplitude of each feature position of the second feature map to obtain an up-sampled feature map, and then the corresponding up-sampled mask map is determined through the up-sampled feature map, so that the up-sampled mask map obtained has a higher degree of graphic freedom and a higher degree of graphic recognition, and the obtained plant morphological phenotypic parameter data is more accurate, making the plant phenotypic analysis and research based on the plant morphological phenotypic parameter data more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0018] Figure 1 A flowchart of a method for processing nighttime plant images according to an embodiment of the present application;
[0019] Figure 2 This is an overall architecture diagram of nighttime plant image processing according to an embodiment of the present application;
[0020] Figure 3 For the first feature in the decoder of the embodiment of the present application Figure X Schematic diagram of the process of obtaining the upsampled feature map Z;
[0021] Figure 4 A comparison diagram of a conventional upsampling operator and a deformable upsampling operator for a foreground image according to an embodiment of the present application;
[0022] Figure 5A comparison diagram of a conventional upsampling operator and a deformable upsampling operator for a background image according to an embodiment of the present application;
[0023] Figure 6 A schematic diagram showing the color characteristics of plants according to an embodiment of the present application;
[0024] Figure 7 A schematic diagram showing the morphological characteristics of plants according to an embodiment of the present application;
[0025] Figure 8 A schematic diagram showing plant texture features according to an embodiment of the present application;
[0026] Fig. 9 A schematic diagram showing a nighttime plant image according to an embodiment of the present application;
[0027] Fig.10 A schematic diagram of a manually labeled pseudo-truth value mask according to an embodiment of the present application;
[0028] Fig.11 For the embodiments of this application Fig. 9 A schematic diagram of the image segmentation result (i.e., up-sampled feature map) of a nighttime plant image;
[0029] Fig.12 This is a result diagram of plant phenotype monitoring of rice throughout its growth period according to an embodiment of the present application;
[0030] Fig.13 This is a structural block diagram of a nighttime plant image processing device according to an embodiment of the present application;
[0031] Fig.14 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0032] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0033] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0034] The nighttime plant image processing method of the embodiment of the present application can be applied to various plants including wheat, rice, millet, buckwheat, etc., and is particularly applicable to the study of the morphological changes of rice in different growth stages.
[0035] The following is an analysis of the image phenotyping of rice:
[0036] As an important food crop, it is of great significance to monitor and analyze the growth status of rice during its growth period. However, due to the large morphological changes of rice plants in different growth periods and the complex real field environment, especially under night lighting conditions, image segmentation methods are difficult to accurately process rice images of different scales, postures, perspectives and deformations. Therefore, there is an urgent need for an image segmentation technology that can adapt to these changes to improve the accuracy of segmentation and parameter extraction.
[0037] As an application scenario of visual dense prediction, the core of image segmentation is to accurately predict each pixel, which requires the model to capture subtle features and complex structures in the image.
[0038] The typical image segmentation architecture is mainly based on the form of encoder and decoder:
[0039] First, the input is encoded by the encoder, and then the features are aggregated and decoded by the decoder's feature fusion module and decoder head module to obtain the final result. For most image segmentation tasks, the final output of the network is usually a result with the same resolution as the input, which puts a great test on the ability of the network's feature upsampling module. Therefore, as the model depth increases and the application scenarios become more complicated, how to effectively upsample low-resolution feature maps to restore high-resolution output has become one of the key issues in improving the performance of image segmentation tasks.
[0040] General upsampling methods, such as nearest neighbor interpolation and bilinear interpolation, are simple and efficient, but they usually lead to blurred edges and loss of details in the upsampling results, making it difficult to meet the needs of high-quality image reconstruction. In order to overcome these limitations, researchers have proposed a variety of upsampling operators based on deep learning, such as deconvolution (transposed convolution) and PixelShuffle. These methods can adaptively adjust the upsampling process through the learned upsampling kernel, thereby better restoring image details.
[0041] In recent years, the research on dynamic feature upsampling operators has gradually become a focus. Such operators dynamically adjust the upsampling kernel so that it can adaptively change according to the content of the input features, thereby achieving a more accurate and detailed upsampling effect. For example, CARAFE (Content-Aware ReAssembly of FEatures) uses a learnable kernel generation network to dynamically generate upsampling kernels to adapt to different contents in the feature map. The A2U operator designs a new guided upsampling operator based on the encoder feature affinity, which has significant effects in the field of image matting. These content-aware dynamic upsampling operators have shown significant advantages in maintaining image edges and texture details.
[0042] Although these methods have made some progress, how to maintain the computational efficiency and adaptability of the algorithm while achieving efficient feature upsampling, as well as how to further improve the upsampling quality, are still urgent issues to be solved. In addition, most high-performance dynamic upsampling operators rely on high-resolution encoding features, which may be difficult to obtain in general agricultural application scenarios, limiting their versatility and scope of application.
[0043] Therefore, it is necessary to propose a simple and efficient upsampling operator that can be plugged and played into the segmentation framework to provide high-precision segmentation results, so as to adapt to complex agricultural scenarios and obtain plant phenotypes with high throughput.
[0044] The embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0045] The nighttime plant image processing method proposed in the embodiment of the present application is as follows: Figure 1 As shown, including:
[0046] Step 101: Encode the nighttime plant image to obtain a first feature map.
[0047] When implementing it, Figure 2As shown, a nighttime plant image (for example, with a size of 2h×2w×c, where 2h is the image height, 2w is the image width, and c is the number of feature channels) is input into the encoder as an input picture, and encoding processing of feature maps with resolutions of 1 / 2, 1 / 4, and 1 / 8 is performed to obtain a first feature map.
[0048] That is, the encoding is performed at a low resolution to obtain a low-resolution first feature map (for example, a feature map with a size of h×w×c Figure X ).
[0049] Step 102, after performing an offset process based on the first feature map, up-sampling is performed using dynamic features in the form of a deformation kernel to obtain a second feature map.
[0050] In specific implementation, step 102 is performed by using an encoder, wherein the dynamic features in the form of a deformable kernel belong to the dynamic features in the form of a deformable kernel. The dynamic feature upsampling algorithm includes two key components: a deformable upsampling kernel and a dynamic modulation module.
[0051] In some embodiments, step 102 includes:
[0052] Step 1021: Use a fully connected layer to perform interpolation upsampling processing on the first feature map to obtain an intermediate feature map.
[0053] When implementing it, Figure 3 As shown, the decoder first passes a fully connected layer to the first feature map (i.e., Figure 3 The input feature map in the image is interpolated and up-sampled (for example, bilinear difference up-sampling) to obtain an intermediate feature map FC(X) (for example, with a size of 2h×2w×c), where FC represents a fully connected layer.
[0054] Step 1022: Perform feature transformation processing on the intermediate feature map according to the first convolutional layer to obtain an offset field, and determine a deformation upsampling kernel based on the offset field.
[0055] In a specific implementation, the first convolution layer is a two-dimensional convolution layer Conv2D, and the offset field B is obtained through an additional two-dimensional convolution layer (ie, Figure 4 and Figure 5 The conventional upsampling operator in (X) is learned from the intermediate feature map, B = Conv2D(FC(X)).
[0056] For example, the offset field B is a graph of size 2h×2w×9c, and each feature point in the offset field corresponds to a two-dimensional offset.
[0057] In some embodiments, step 1022 includes:
[0058] Step 10221, using the first convolutional layer to perform learning and training according to the intermediate feature map, obtain the offset corresponding to each feature point in the intermediate feature map, and combine the offset matrix composed of all offsets into the offset field.
[0059] In specific implementation, consider a certain feature point (i, j) on the offset field B (for example, B i,j (1×1×9c)), each feature point corresponds to a two-dimensional offset, for better explanation, it is rearranged into b k The (3×3×c) offset matrix constitutes the offset field B.
[0060] Step 10222, using the initial convolution kernel to couple with the offset matrix of the corresponding position in the offset field, to obtain the initial deformation upsampling kernel of the corresponding position.
[0061] Step 10223, all initial deformation upsampling kernels are combined to obtain the deformation upsampling kernel (ie, Figure 4 and Figure 5 The deformable upsampling operator in .
[0062] In the specific implementation, four initial convolution kernels {w k |k=0,1,2,3} respectively correspond to the offset matrix {b k |k=0, 1, 2, 3} are coupled to obtain the initial deformation upsampling kernel at the corresponding position. All initial deformation upsampling kernels are combined as the deformation upsampling kernel.
[0063] In this way, the receptive field of the image can be adaptively adjusted based on the deformable upsampling kernel, so that the model containing the deformable upsampling kernel can better handle the different scales, postures, perspectives and deformations of objects in the image, and increase the modeling ability of conventional convolution for scale, posture, perspective and deformation, better cover the entire object for the foreground class, and include more contextual information for the background class.
[0064] Step 1023: Process the first feature map using the deformed upsampling kernel to obtain a second feature map.
[0065] In some embodiments, step 1023 includes:
[0066] Step 10231: divide the two-dimensional grid corresponding to the first feature map into multiple areas (for example, 4 areas).
[0067] Step 10232, for each region k (for example, k = 0, 1, 2, 3), execute: determine the coordinates of the neighborhood center point of the region, and the deformation upsampling kernel corresponding to each position point in the region; input the coordinates of the neighborhood center point and the part of the first feature map corresponding to the region into the deformation upsampling kernel corresponding to the region to obtain the second feature map.
[0068] In the specific implementation, the coordinates of the center point of each area are determined as {p k |k=0,1,2,3},where P n is a feature point on the two-dimensional graphic grid, and p k The corresponding offset in the deformed upsampling kernel is {b k |k=0,1,2,3},determine the initial convolution kernel w k At the feature point P n The value w on k (p n ).
[0069] The corresponding outputs for each region are:
[0070]
[0071]
[0072] The Y(p 0 )、Y(p 1 )、Y(p 2 )、Y(p 3 ) is combined as the second feature map Y.
[0073] Step 103: Use a dynamic modulation module to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map.
[0074] The deformed upsampling kernel obtained in step 102 can only adjust the receptive field of the feature, but cannot adjust the amplitude of the feature. In order to adjust the amplitude of the feature, the following technical solution is adopted:
[0075] In some embodiments, after step 1021, step 103 further includes:
[0076] Step 1031: Use the second convolutional layer to perform feature transformation processing on the intermediate feature map to obtain a modulation field, and determine the dynamic modulation module based on the modulation field.
[0077] In a specific implementation, the second convolution layer may be a two-dimensional convolution layer, and a modulation field can be obtained after the intermediate feature map passes through the second convolution layer, and each feature point in the modulation field corresponds to a modulation value.
[0078] For example: for the intermediate feature map FC(X) of 2h×2w×c, after the feature transformation by the second convolutional layer Conv2D, the modulation field M is obtained. The size of the modulation field M is 2h×2w×c, and the corresponding formula is: M=Conv2D(FC(X)).
[0079] Step 1032, using the modulation field in the dynamic modulation module to perform multiplication processing with the second feature map, complete the amplitude adjustment of each feature position of the second feature map, and obtain the up-sampled feature map.
[0080] In specific implementation, the modulation field M in the dynamic modulation module is used to modulate the second feature map Y to obtain the upsampled feature map Z, specifically: Z = Y⊙M, ⊙ represents an element-by-element multiplication operation. In this way, the amplitude of the feature point (i.e., the feature position) corresponding to each sampling position in the second feature map Y can be adjusted.
[0081] This increases the degree of freedom of the entire processing model (i.e., the processing model for implementing the method of the present application). Based on the upsampled feature map, for some irrelevant sampling points, the weights can be learned to be 0, thereby improving the segmentation quality and obtaining a segmentation mask. Fig. 9 As shown in (a), the segmentation mask is used to perform segmentation processing to obtain the image segmentation result (i.e., the up-sampled mask map) as shown in Fig.11 As shown in the figure, the pseudo-truth mask (such as Fig.10 The processing model is supervised by the trained model (as shown in Figure 2), thereby continuously improving the accuracy of the processing model.
[0082] Step 104, determining an up-sampled mask image corresponding to the up-sampled feature image, performing plant phenotype extraction on the up-sampled mask image to obtain plant morphological phenotype parameter data, and outputting the plant morphological phenotype parameter data.
[0083] In a specific implementation, the corresponding plant phenotypic parameter data includes at least one of the following: plant color characteristics, plant morphological characteristics, and plant texture characteristics.
[0084] Among them, the process of determining the color characteristics of plants is as follows:
[0085] Step A1, separating a green channel from the nighttime plant image, and determining a mask area corresponding to the up-sampling mask image.
[0086] Step A2: applying the mask area to the green channel, removing pixels in other areas of the green channel outside the mask area, obtaining a green mask image, and calculating the average green component G based on the green mask image. avg .
[0087] In specific implementation, the average green component G avg is the average value of the green channel of the image within the mask area.
[0088] The formula is: Among them, G i is the green channel value of the ith pixel in the mask area, and n is the total number of pixels in the mask area.
[0089] Alternatively, in step A3, the spatial hue (HSV spatial hue) of the nighttime plant image in the mask area is determined, and the average value of the spatial hue is calculated to obtain the average hue H avg .
[0090] In the specific implementation, the night plant image is converted from BGR to HSV color space, and then the hue channel (i.e., spatial hue) in the HSV color space is separated using the mask area, and the average value H of the hue channel in the mask area is calculated. avg , the formula is as follows:
[0091]
[0092] Among them, H i is the hue value of the i-th pixel in the spatial hue within the mask area (expressed in degrees, 0 to 360), and n is the total number of pixels in the mask area.
[0093] Alternatively, in step A4, the number of red pixels G, green pixels R and blue pixels B of the night plant image in the mask area are determined, and ExG=2*GRB and ExR=1.4*RB are calculated, where ExG (Excess Green) is the number of super green pixels and ExR (Excess Red) is the number of super red pixels.
[0094] Step A5, determining the number of green leaf pixels satisfying ExG>0.01×(R+G+B) and ExR<0.4×(R+G+B) in the nighttime plant image within the mask area, and determining the green leaf area according to the number of green leaf pixels.
[0095] Step A6, taking at least one of the average green component, average hue and green leaf area as a plant color feature (eg Figure 6 shown).
[0096] Through the above scheme, the color characteristics of the plant can be obtained, so that the growth status of the plant and whether the plant has disease or insect infestation, etc. can be determined according to the color characteristics.
[0097] In some embodiments, the process of determining the plant morphological characteristics includes:
[0098] Step B1, determining a minimum bounding rectangle of the up-sampled mask image, and determining a plant canopy width and / or a plant canopy length according to the minimum bounding rectangle.
[0099] In a specific implementation, the minimum bounding rectangle of the up-sampled mask image is determined, and the coordinates of the four vertices of the minimum bounding rectangle are obtained to calculate the plant canopy width and / or the plant canopy length.
[0100] Alternatively, in step B2, the plant contour in the up-sampled mask image is determined, the contour arc length is calculated according to the plant contour, and the contour arc length is used as the perimeter P of the plant.
[0101] In specific implementation, the calculation formula of the plant perimeter P is as follows:
[0102]
[0103] Among them, (x i ,y j ) is the coordinate of the i-th pixel point on the plant outline, and n is the number of pixels on the plant outline.
[0104] Alternatively, step B3, determining the plant outline in the up-sampled mask image, determining the convex hull outline according to the plant outline, and calculating the convex hull area A according to the convex hull outline h ull .
[0105] In specific implementation, the calculation formula of the convex hull area is:
[0106]
[0107] Among them, (x i ,y i ) is the coordinate of the i-th pixel point on the convex hull contour, and m is the number of pixels on the convex hull contour.
[0108] Step B4, taking at least one of the plant canopy width, plant canopy length, plant perimeter and convex hull area as a plant morphological feature (such as Figure 7 shown).
[0109] Through the above scheme, more accurate plant morphological characteristics can be obtained, so that the morphology of the plant can be known, and the current growth stage and growth status of the plant can be judged according to the plant morphology.
[0110] In some embodiments, the process of determining plant texture features includes:
[0111] Step C1, determining a gray level co-occurrence matrix (GLCM) corresponding to the up-sampled mask image.
[0112] In specific implementation, GLCM describes the spatial relationship between the grayscale values of pixels in an image.
[0113] Step C2, normalizing the gray level co-occurrence matrix to obtain a normalized gray level co-occurrence matrix, and calculating an entropy value E according to the normalized gray level co-occurrence matrix.
[0114] In specific implementation, the formula for the entropy value E is:
[0115] E=-∑ i,j p(i, j)log 2 p(i, j);
[0116] Where p(i, j) is the pixel in the i-th row and j-th column in the normalized gray-level co-occurrence matrix.
[0117] Alternatively, in step C3, each pixel point in the gray level co-occurrence matrix is determined, and a contrast ratio Contrast is calculated according to each pixel point in the gray level co-occurrence matrix.
[0118] In specific implementation, the calculation formula of contrast Contrast is: Contrast = ∑ i,j (ij) 2 p(i, j);
[0119] Where p(i, j) is the pixel in the i-th row and j-th column in the gray-level co-occurrence matrix.
[0120] Alternatively, in step C4, a pixel mean value and a pixel standard deviation of each pixel in the gray level co-occurrence matrix are determined, and a correlation Correlation is calculated according to the pixel mean value and the pixel standard deviation.
[0121] In specific implementation, the calculation formula of correlation is:
[0122]
[0123] Among them, μ i and μ j are the pixel means of the i-th row and j-th column in the gray-level co-occurrence matrix, σ i and σ j are the standard deviations of the pixels in the i-th row and j-th column in the gray-level co-occurrence matrix.
[0124] Step C5, taking at least one of the entropy value, contrast and correlation as a plant texture feature (such as Figure 8 shown).
[0125] Through the above scheme, more accurate plant texture features can be obtained, and the growth indicators of plants can be monitored based on these texture features, and then problems in the plant growth process can be discovered in time and handled in time based on the monitoring results.
[0126] The method of the present application is mainly aimed at monitoring rice at different periods, especially the dynamic detection effect of rice growth period at night is outstanding.
[0127] Image segmentation (ie, obtaining corresponding up-sampling mask images) and plant phenotype extraction (ie, plant morphological phenotype parameter data) are performed on night rice images at different development stages at different days after sowing.
[0128] Specific as Fig.12 As shown, each column shows the original image, segmentation mask, foreground, and phenotypic details, showing in detail the progress and phenotypic changes throughout the growth stage. This comprehensive monitoring can analyze and understand the dynamic growth pattern of rice in detail, accurately extract the main growth indicators such as rice canopy width and length, and provide guidance for rice breeding.
[0129] As shown in the figure, the above step 101 is performed using an encoder, and steps 102 and 103 are performed using a decoder, wherein a processing model is provided in the decoder to complete the decoding process to obtain an upsampled feature map.
[0130] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.
[0131] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0132] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a nighttime plant image processing device.
[0133] refer to Fig.13 , the device comprises:
[0134] The encoding module 201 is configured to encode the nighttime plant image to obtain a first feature map;
[0135] An offset processing module 202 is configured to perform an offset process based on the first feature map, and then perform an upsampling process using a dynamic feature in the form of a deformation kernel to obtain a second feature map;
[0136] A dynamic modulation module 203 is configured to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map;
[0137] The plant phenotype extraction module 204 is configured to determine an up-sampled mask image corresponding to the up-sampled feature image, perform plant phenotype extraction on the up-sampled mask image, obtain plant morphological phenotype parameter data, and output the plant morphological phenotype parameter data.
[0138] In some embodiments, the offset processing module 202 is specifically configured to:
[0139] Using a fully connected layer to perform interpolation upsampling processing on the first feature map to obtain an intermediate feature map;
[0140] Performing feature transformation processing on the intermediate feature map according to the first convolutional layer to obtain an offset field, and determining a deformation upsampling kernel based on the offset field;
[0141] The first feature map is processed by the deformed upsampling kernel to obtain a second feature map.
[0142] In some embodiments, the offset processing module 202 is further configured to:
[0143] Using the first convolutional layer to perform learning and training according to the intermediate feature map, obtain the offset corresponding to each feature point in the intermediate feature map, and combine the offset matrix composed of all offsets into the offset field;
[0144] The initial convolution kernel is coupled with the offset matrix of the corresponding position in the offset field to obtain the initial deformation upsampling kernel of the corresponding position;
[0145] All initial deformed upsampling kernels are combined to obtain the deformed upsampling kernel.
[0146] In some embodiments, the offset processing module 202 is further configured to:
[0147] Dividing the two-dimensional grid corresponding to the first feature map into a plurality of regions;
[0148] For each region, the following steps are performed: determining the coordinates of the neighborhood center point of the region and the deformed upsampling kernel corresponding to each position point in the region; inputting the coordinates of the neighborhood center point and the portion of the first feature map corresponding to the region into the deformed upsampling kernel corresponding to the region to obtain the second feature map.
[0149] In some embodiments, the dynamic modulation module 203 is specifically configured as follows:
[0150] After the first feature map is interpolated and up-sampled by the fully connected layer to obtain an intermediate feature map, the intermediate feature map is subjected to feature transformation processing by the second convolutional layer to obtain a modulation field, and the dynamic modulation module 203 is determined based on the modulation field;
[0151] The modulation field in the dynamic modulation module 203 is used to perform multiplication processing with the second feature map to complete the amplitude adjustment of each feature position of the second feature map to obtain an up-sampled feature map.
[0152] In some embodiments, the plant phenotype extraction module 204 is specifically configured to:
[0153] Separating a green channel from the nighttime plant image and determining a mask area corresponding to the up-sampled mask image;
[0154] Applying the mask area to the green channel, removing pixels in other areas of the green channel outside the mask area to obtain a green mask image, and calculating an average green component based on the green mask image; or,
[0155] Determine the spatial hue of the nighttime plant image in the mask area, and calculate the average value of the spatial hue to obtain the average hue; or,
[0156] Determine the number of red pixels G, the number of green pixels R, and the number of blue pixels B of the nighttime plant image in the mask area, and calculate ExG=2*GRB and ExR=1.4*RB, where ExG is the number of super green pixels and ExR is the number of super red pixels;
[0157] Determine the number of green leaf pixels corresponding to ExG>0.01×(R+G+B) and ExR<0.4×(R+G+B) in the nighttime plant image within the mask area, and determine the green leaf area according to the number of green leaf pixels;
[0158] At least one of the average green component, average hue and green leaf area is used as a plant color feature.
[0159] In some embodiments, the plant phenotype extraction module 204 is specifically configured to:
[0160] Determine a minimum bounding rectangle of the up-sampled mask image, and determine a plant canopy width and / or a plant canopy length according to the minimum bounding rectangle; or,
[0161] Determine the plant contour in the up-sampled mask image, calculate the contour arc length according to the plant contour, and use the contour arc length as the perimeter of the plant; or,
[0162] Determine a plant outline in the up-sampled mask image, determine a convex hull outline according to the plant outline, and calculate a convex hull area according to the convex hull outline;
[0163] At least one of the plant canopy width, plant canopy length, plant perimeter and convex hull area is used as a plant morphological feature.
[0164] In some embodiments, the plant phenotype extraction module 204 is specifically configured to:
[0165] Determining a gray level co-occurrence matrix corresponding to the up-sampled mask image;
[0166] Normalizing the gray-level co-occurrence matrix to obtain a normalized gray-level co-occurrence matrix, and calculating the entropy value according to the normalized gray-level co-occurrence matrix; or,
[0167] Determine each pixel point in the gray level co-occurrence matrix, and calculate the contrast according to each pixel point in the gray level co-occurrence matrix;
[0168] Determine the pixel mean of each pixel point in the gray level co-occurrence matrix, and the pixel standard deviation of each pixel point, and calculate the correlation according to the pixel mean and the pixel standard deviation;
[0169] At least one of the entropy value, contrast and correlation is used as a plant texture feature.
[0170] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0171] The device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0172] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.
[0173] Fig.14A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0174] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0175] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0176] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0177] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0178] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).
[0179] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0180] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0181] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.
[0182] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0183] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0184] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.
[0185] It is understandable that before using the technical solutions of each embodiment of the present application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.
[0186] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.
[0187] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0188] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of this application. Other methods that meet relevant laws and regulations may also be applied to the implementation method of this application.
[0189] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0190] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.
[0191] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0192] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.
Claims
1. A method for processing nighttime plant images, characterized in that: include: Encoding the nighttime plant image to obtain a first feature map; After performing an offset process based on the first feature map, upsampling is performed using dynamic features in the form of a deformation kernel to obtain a second feature map; Using a dynamic modulation module to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map; An up-sampling mask image corresponding to the up-sampling feature image is determined, plant phenotype extraction is performed on the up-sampling mask image to obtain plant morphological phenotype parameter data, and the plant morphological phenotype parameter data is output.
2. The method according to claim 1, characterized in that After performing the offset processing based on the first feature map, upsampling processing is performed using the dynamic features in the form of a deformation kernel to obtain a second feature map, including: Using a fully connected layer to perform interpolation upsampling processing on the first feature map to obtain an intermediate feature map; Performing feature transformation processing on the intermediate feature map according to the first convolutional layer to obtain an offset field, and determining a deformation upsampling kernel based on the offset field; The first feature map is processed by the deformed upsampling kernel to obtain a second feature map.
3. The method according to claim 2, characterized in that The step of performing feature transformation processing on the intermediate feature map according to the first convolutional layer to obtain an offset field, and determining a deformation upsampling kernel based on the offset field includes: Using the first convolutional layer to perform learning and training according to the intermediate feature map, obtain the offset corresponding to each feature point in the intermediate feature map, and combine the offset matrix composed of all offsets into the offset field; The initial convolution kernel is coupled with the offset matrix of the corresponding position in the offset field to obtain the initial deformation upsampling kernel of the corresponding position; All initial deformation upsampling kernels are combined to obtain the deformation upsampling kernel.
4. The method according to claim 2, characterized in that: The step of processing the first feature map by using the deformed upsampling kernel to obtain a second feature map includes: Dividing the two-dimensional grid corresponding to the first feature map into a plurality of regions; For each region, execute: Determine the coordinates of the neighborhood center point of the region and the deformation upsampling kernel corresponding to each position point in the region; The coordinates of the center point of the neighborhood and the portion of the first feature map corresponding to the region are input into the deformation upsampling kernel corresponding to the region to obtain the second feature map.
5. The method according to claim 2, characterized in that: After the first feature map is interpolated and up-sampled by using the fully connected layer to obtain an intermediate feature map, the method further includes: Using a second convolutional layer to perform feature transformation processing on the intermediate feature map to obtain a modulation field, and determining the dynamic modulation module based on the modulation field; The step of using a dynamic modulation module to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map includes: The modulation field in the dynamic modulation module is used to perform multiplication processing with the second feature map to complete the amplitude adjustment of each feature position of the second feature map to obtain an up-sampled feature map.
6. The method according to claim 1, characterized in that Plant morphological phenotypic parameter data include: plant color characteristics; Extracting plant phenotypes from the upsampled mask image to obtain plant morphological phenotype parameter data, and outputting the plant morphological phenotype parameter data, including: Separating a green channel from the nighttime plant image and determining a mask area corresponding to the up-sampled mask image; Applying the mask area to the green channel, removing pixels in other areas of the green channel outside the mask area to obtain a green mask image, and calculating an average green component based on the green mask image; or, Determine the spatial hue of the nighttime plant image in the mask area, and calculate the average value of the spatial hue to obtain the average hue; or, Determine the number of red pixels G, the number of green pixels R, and the number of blue pixels B of the nighttime plant image in the mask area, and calculate ExG=2*GRB and ExR=1.4*RB, where ExG is the number of super green pixels and ExR is the number of super red pixels; Determine the number of green leaf pixels corresponding to ExG>0.01×(R+G+B) and ExR<0.4×(R+G+B) in the nighttime plant image within the mask area, and determine the green leaf area according to the number of green leaf pixels; At least one of the average green component, average hue and green leaf area is used as a plant color feature.
7. The method according to claim 1, characterized in that Plant morphological phenotypic parameter data include: plant morphological characteristics; Extracting plant phenotypes from the upsampled mask image to obtain plant morphological phenotype parameter data, and outputting the plant morphological phenotype parameter data, including: Determine a minimum bounding rectangle of the up-sampled mask image, and determine a plant canopy width and / or a plant canopy length according to the minimum bounding rectangle; or, Determine the plant contour in the up-sampled mask image, calculate the contour arc length according to the plant contour, and use the contour arc length as the perimeter of the plant; or, Determine a plant outline in the up-sampled mask image, determine a convex hull outline according to the plant outline, and calculate a convex hull area according to the convex hull outline; At least one of the plant canopy width, plant canopy length, plant perimeter and convex hull area is used as a plant morphological feature.
8. The method according to claim 1, characterized in that Plant morphological phenotypic parameter data include: plant texture characteristics; Extracting plant phenotypes from the upsampled mask image to obtain plant morphological phenotype parameter data, and outputting the plant morphological phenotype parameter data, including: Determining a gray level co-occurrence matrix corresponding to the up-sampled mask image; Normalizing the gray-level co-occurrence matrix to obtain a normalized gray-level co-occurrence matrix, and calculating the entropy value according to the normalized gray-level co-occurrence matrix; or, Determine each pixel point in the gray level co-occurrence matrix, and calculate the contrast according to each pixel point in the gray level co-occurrence matrix; Determine the pixel mean of each pixel point in the gray level co-occurrence matrix, and the pixel standard deviation of each pixel point, and calculate the correlation according to the pixel mean and the pixel standard deviation; At least one of the entropy value, contrast and correlation is used as a plant texture feature.
9. A nighttime plant image processing device, characterized in that: include: An encoding module is configured to encode the nighttime plant image to obtain a first feature map; an offset processing module, configured to perform an offset process based on the first feature map, and then perform an upsampling process using a dynamic feature in the form of a deformation kernel to obtain a second feature map; A dynamic modulation module is configured to perform amplitude adjustment processing on each feature position of the second feature map to obtain an upsampled feature map; The plant phenotype extraction module is configured to determine an up-sampled mask image corresponding to the up-sampled feature image, perform plant phenotype extraction on the up-sampled mask image, obtain plant morphological phenotype parameter data, and output the plant morphological phenotype parameter data.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
High-resolution ship detection method based on cross attention and dynamic sampling
CN121392262A
High-resolution ship detection method based on cross-attention and dynamic sampling
CN121392262B
