Deep learning-based plant insect disease image identification method and system

Abstract

The invention provides a plant insect disease image identification method and system based on deep learning, and belongs to the technical field of deep learning, and the method comprises the steps: firstly obtaining a multi-source image set of a plurality of growth stages of a plant, including the whole, leaf and stem images of the plant, the growth stage time, and the shooting angle identification information; constructing a plant image time sequence association network to perform organ-level time sequence alignment processing to obtain an aligned image set and an organ association coefficient, performing cross-growth-stage focus feature association through the association coefficient to obtain a cross-stage focus association set and time sequence evolution information, and performing cross-growth-stage focus feature association; and inputting the association set into an exclusive deep learning model for feature aggregation and type matching to obtain a preliminary result, and finally performing organ adaptation verification on the preliminary result in combination with time sequence evolution information to generate a final identification result and a focus organ time sequence evolution atlas, thereby improving the accuracy and reliability of plant insect disease identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a plant pest and disease image recognition method and system based on deep learning. BACKGROUND

[0002] In the field of agriculture, accurate identification of plant pests and diseases is crucial for ensuring crop yield and quality. Traditional methods of plant pest and disease identification rely mainly on manual observation and experience-based judgment. Such methods are not only inefficient, but also susceptible to the subjective factors of the observers, resulting in low accuracy. With the development of image recognition technology, image-based plant pest and disease identification methods have emerged.

[0003] However, existing image-based identification methods mostly analyze plant images of a single growth stage, ignoring the important factor that the appearance characteristics and pest and disease manifestations of plants change at different growth stages. Plants at different growth stages have different overall plant morphologies, leaf and stem physiological characteristics, and pest and disease manifestations and impact levels also differ at each stage. Relying solely on single-stage images for identification makes it difficult to fully and accurately grasp the characteristics and development patterns of pest and disease, leading to misjudgments and omissions.

[0004] In addition, existing methods often do not adequately consider the relevance between different plant organs (such as leaves and stems) and the temporal evolution of pest and disease on different organs. Plant pest and disease often spreads and spreads between different organs, and its development process is a dynamic temporal change process. Lack of analysis of the above relevance and temporal evolution makes the identification results lack systematicness and comprehensiveness. SUMMARY

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a plant pest and disease image recognition method based on deep learning, which comprises:

[0006] obtaining a multi-source image set of plants at multiple growth stages, the multi-source image set containing overall plant images, leaf local images, stem local images, and growth stage time identifiers and shooting angle identifier information for each growth stage;

[0007] constructing a plant image temporal correlation network based on the growth stage time identifiers and shooting angle identifier information, performing organ-level temporal alignment processing on the multi-source image set to obtain an organ-level temporal alignment image set and an inter-organ correlation coefficient;

[0008] performing cross-growth-stage lesion feature correlation processing on the organ-level temporal alignment image set through the organ correlation coefficient to obtain a cross-stage lesion correlation set and temporal evolution information of lesions on different organs;

[0009] The cross-stage lesion association set is input into a preset plant pest disease specific deep learning model for feature aggregation and type matching processing, to obtain lesion aggregation features and a preliminary pest disease type result;

[0010] The preliminary pest disease type result is subjected to organ adaptation verification processing in combination with time sequence evolution information of the lesion on different organs, to generate a final plant pest disease recognition result and a lesion organ time sequence evolution graph.

[0011] In still another aspect, an embodiment of the present application further provides a plant pest disease image recognition system based on deep learning, comprising:

[0012] A processor; a machine readable storage medium for storing machine executable instructions of the processor; wherein the processor is configured to execute the above-described plant pest disease image recognition method based on deep learning by executing the machine executable instructions.

[0013] In still another aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising machine executable instructions stored in a computer readable storage medium, a processor of a computer device reading the machine executable instructions from the computer readable storage medium, the processor executing the machine executable instructions, so that the computer device executes the above-described plant pest disease image recognition method based on deep learning.

[0014] Based on the above aspects, by acquiring a multi-source image set of plants at multiple growth stages, image information at different levels such as the whole plant, the local leaf, and the local stem, and growth stage time and shooting angle identifiers are covered, and then a plant image time sequence association network is constructed based on the growth stage time and the shooting angle identifier, the organ-level time sequence alignment image set and the organ correlation coefficient between images are accurately acquired by performing organ-level time sequence alignment processing on the multi-source image set, effectively solving the image alignment problem under different growth stages and shooting angles. Next, the organ-level time sequence alignment image set is processed by the organ correlation coefficient to associate the cross-growth-stage lesion features, which can deeply mine the feature changes of the lesion at different growth stages and the time sequence evolution information on different organs. The cross-stage lesion association set is input into a preset plant pest and disease specific deep learning model for feature aggregation and type matching processing, which can fully utilize the powerful feature extraction and classification ability of the deep learning model to obtain accurate lesion aggregation features and preliminary pest and disease type results. Finally, the organ adaptation verification processing is performed on the preliminary pest and disease type results combined with the time sequence evolution information of the lesion on different organs, which further improves the accuracy and reliability of the recognition results, and generates the final plant pest and disease recognition results and the lesion organ time sequence evolution atlas, providing comprehensive, accurate, and systematic decision-making basis for the prevention and control of plant pests and diseases in agricultural production, effectively improving the intelligent level of agricultural production and the effect of pest and disease control. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is the execution flow diagram of the plant pest and disease image recognition method based on deep learning provided by an embodiment of the present application.

[0016] Figure 2 is a schematic diagram of exemplary hardware and software components of the plant pest and disease image recognition system based on deep learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application will be described in detail below with reference to the accompanying drawings, Figure 1 is a flow diagram of the plant pest and disease image recognition method based on deep learning provided by an embodiment of the present application, and the plant pest and disease image recognition method based on deep learning will be described in detail below.

[0018] Step S110: Acquire a multi-source image set of plants at multiple growth stages, the multi-source image set containing whole plant images, local leaf images, local stem images, and growth stage time identifier and shooting angle identifier information at each growth stage.

[0019] In this embodiment, wheat is taken as the target plant for pest and disease identification. Multi-source images are obtained from five growth stages of wheat, i.e., seed germination stage, tillering stage, jointing stage, heading stage and grain filling stage. For each growth stage, a professional camera is used to take overall plant images at different positions and angles in the field, which can show the overall morphology of the wheat plant at the growth stage, including plant height, tiller number and other information. At the same time, macro lens is used to take local images of leaves, focusing on capturing details on the surface of the leaves, such as whether there are spots, curling and other conditions. Local images of stems are also taken to observe whether there are tunnels, discoloration and other phenomena on the stems. Each image is attached with corresponding growth stage time identifier, for example, the seed germination stage corresponds to the shooting time in the early of a certain month of a certain year, the tillering stage corresponds to the shooting time in the middle of a certain month of the same year, and so on, as well as shooting angle identifier information, such as 45 degrees front, 30 degrees side, top view, etc. The above images and identifier information together constitute a multi-source image set.

[0020] When collecting these image data, if it involves private sensitive data such as geographical information of the shooting area, data desensitization technology is used to convert the specific coordinates in the geographical position information into fuzzy area identifier, and the image data is stored in encrypted form. Multiple identity verifications are required when accessing to prevent privacy leakage.

[0021] Step S120: Based on the growth stage time identifier and the shooting angle identifier information, a plant image time sequence association network is constructed, and organ-level time sequence alignment processing is performed on the multi-source image set to obtain an organ-level time sequence alignment image set and an organ correlation coefficient between images.

[0022] Step S121: Extract the growth stage time identifier and the shooting angle identifier information of each image in the multi-source image set, and perform primary sorting on all images according to the growth stage sequence, and perform secondary sorting within the same growth stage according to the shooting angle identifier information to obtain an initial time sequence angle image sequence.

[0023] For the above multi-source image set of wheat, first, the growth stage time identifier and the shooting angle identifier information of each image are extracted. All images are primarily sorted according to the sequence of seed germination stage, tillering stage, jointing stage, heading stage and grain filling stage. Within the same growth stage, for example, the tillering stage, secondary sorting is performed according to the shooting angle identifier information, such as first 45 degrees front, then 30 degrees side, and finally top view. Through the above sorting method, the originally disordered multi-source images are sorted into an ordered initial time sequence angle image sequence, so that the images have a correlation in time and angle, which prepares for the subsequent organ-level time sequence alignment processing.

[0024] Step S122: Extract the plant organ contour features of each image in the initial time sequence angle image sequence, the plant organ contour features including stem contour features, leaf contour features and stem branch contour features, the stem contour features including stem contour point coordinates and contour curvature change values, the leaf contour features including leaf contour point coordinates, leaf vein contour direction and leaf edge serration density, and the stem branch contour features including branch contour point coordinates and branch connection angle with the main stem.

[0025] Step S1221: Perform color space conversion processing on each image in the initial time sequence angle image sequence, extract the hue channel image, and perform threshold segmentation processing on the hue channel image to divide the hue channel image into plant foreground area and background area, to obtain a binary image.

[0026] For the wheat image in the initial time sequence angle image sequence, the common RGB color space is converted to HSV color space, and then the hue channel image is extracted. Since the wheat plant is different in hue from the background (such as soil, weeds, etc.), by performing threshold segmentation processing on the hue channel image and setting an appropriate threshold range, the area where the wheat plant is located is divided into a plant foreground area, and the background area is excluded, thereby obtaining a binary image. In the binary image, the plant foreground area is white and the background area is black, and the approximate outline of the wheat plant is clearly separated.

[0027] Step S1222: After performing morphological closing operation processing on the binary image, perform morphological opening operation processing to obtain an optimized binary image, and perform edge detection processing on the optimized binary image to form an initial contour line segment set.

[0028] The above obtained binary image is first subjected to morphological closing operation processing, which can fill small holes in the image and make the plant foreground area contour more complete. Then, morphological opening operation processing is performed, which can remove small noise points in the image and further optimize the image quality to obtain an optimized binary image. Then, the optimized binary image is processed using an edge detection algorithm to detect the edges of the plant foreground area, which are composed of a series of line segments to form an initial contour line segment set. The above line segments preliminarily outline the contour shape of each organ of the wheat plant.

[0029] Step S1223: Perform contour screening processing on the initial contour line segment set, set screening rules according to the morphological characteristics of the plant organs, screen out stem contour line segments, leaf contour line segments and stem branch contour line segments, perform fitting processing on the screened stem contour line segments, use a cubic polynomial fitting algorithm to fit the line segments into a smooth curve that conforms to the upright shape of the plant stem, extract the sampling point coordinates on the smooth curve as the stem contour point coordinates, calculate the curvature change value at each sampling point as the stem contour curvature change value, and form the stem contour feature.

[0030] According to the morphological characteristics of each organ of the wheat plant, the screening rules are set, and the initial contour line segment set is screened. The wheat stem is usually upright, the contour line segment is relatively long and the trend is relatively stable, and the above characteristics are screened out the stem contour line segment; the leaf contour line segment has a relatively wide spread range, and the edge may have a jagged shape; the stem branch contour line segment is relatively thin, and has a connection point with the stem contour line segment. After screening out the stem contour line segment, a cubic polynomial fitting algorithm is used for fitting, and a smooth curve is obtained, which conforms to the upright shape of the wheat stem. Then, sampling point coordinates are extracted on the smooth curve at fixed intervals, which are the stem contour point coordinates. For each sampling point, the curvature change value is calculated, which reflects the bending degree of the stem contour at that point. The stem contour point coordinates and the corresponding curvature change value are combined to form the stem contour feature.

[0031] Step S1224: Perform closed processing on the screened leaf contour line segment, extend the open line segment to intersect to form a closed contour, extract the sampling point coordinates on the closed contour as the leaf contour point coordinates, extract the contour trend of the main vein and side vein of the leaf by the vein detection algorithm, count the number and distribution density of the jagged line segments on the leaf edge as the leaf edge jagged density, and form the leaf contour feature.

[0032] For the screened leaf contour line segment, since there may be open line segments in the actual shooting and image processing process, closed processing is needed. The two ends of the open line segment are extended according to certain rules until the line segments intersect to form a closed leaf contour. The sampling point coordinates on the closed contour are extracted as the leaf contour point coordinates. Then, the leaf region is processed using the vein detection algorithm to extract the contour trend of the main vein and side vein of the leaf. The main vein is usually the thicker vein in the center of the leaf, and the side vein extends from the main vein to the edge of the leaf. At the same time, the number of jagged line segments on the edge of the leaf is counted, and the distribution density, i.e. the number of jagged line segments per unit length on the edge of the leaf, is calculated. The leaf contour point coordinates, vein contour trend and leaf edge jagged density are combined to form the leaf contour feature.

[0033] Step S1225: fitting the screened stem branch contour line segment, obtaining branch contour point coordinates, calculating the angle between the branch contour line segment and the main stem contour curve at the connection point as the connection angle of the branch and the main stem, and forming the stem branch contour feature.

[0034] The screened stem branch contour line segment is fitted, a similar fitting method is used for the main stem contour line segment, a smooth curve of the branch contour is obtained, and the sampling point coordinates on the curve are extracted as the branch contour point coordinates. Then, through an angle calculation algorithm, the connection point of the branch contour line segment and the main stem contour curve is found, the angle between the tangent direction of the branch contour curve at the connection point and the tangent direction of the main stem contour curve is calculated, and the angle is the connection angle of the branch and the main stem. The branch contour point coordinates and the connection angle are combined to form the stem branch contour feature.

[0035] Step S1226: combining the main stem contour feature, the leaf contour feature, and the stem branch contour feature into the plant organ contour feature of each image, and associating and recording the connection point coordinates of the branch and the main stem and the connection point coordinates of the leaf and the branch.

[0036] The main stem contour feature, the leaf contour feature, and the stem branch contour feature obtained above are integrated together to form the complete plant organ contour feature of each image. At the same time, the connection point coordinates of the branch and the main stem and the connection point coordinates of the leaf and the branch are associated and recorded, which can reflect the positional relationship between the organs of the wheat plant.

[0037] Step S123: taking each image as a network node, taking the similarity of the plant organ contour features of the images of the same shooting angle in adjacent growth stages as the initial edge weight, constructing an initial time sequence association network, and the initial edge weight is calculated by weighting the standardized values of the main stem contour overlap ratio, the leaf contour overlap ratio, and the branch connection angle deviation value, wherein the branch connection angle deviation value is converted into a deviation score comparable to the contour overlap ratio, the higher the main stem contour overlap ratio, the higher the leaf contour overlap ratio, and the higher the branch connection angle deviation score, the larger the initial edge weight.

[0038] Each image of the initial time-series angular image sequence is regarded as a network node. For two images of the same shooting angle of adjacent growth stages, the similarity of the plant organ contour features is calculated, and the similarity is used as the initial edge weight to construct the initial time-series correlation network. When calculating the initial edge weight, first, the stem contour overlap ratio, the leaf contour overlap ratio, and the branch connection angle deviation value are calculated respectively. The stem contour overlap ratio refers to the proportion of the number of overlapping stem contour point coordinates in the total number of contour points in the two images; the leaf contour overlap ratio is the same. The branch connection angle deviation value is the difference between the connection angles of the corresponding branches and the stem in the two images. The difference is converted into a deviation score by mapping the angle difference to a score interval of the same range as the contour overlap ratio. The smaller the angle difference, the higher the deviation score. Then, the three indicators are standardized to be in the same order of magnitude, and then weighted according to certain weights to obtain the initial edge weight. The higher the stem contour overlap ratio, the higher the leaf contour overlap ratio, and the higher the branch connection angle deviation score, the more similar the plant organ contour features of the two images, and the larger the initial edge weight.

[0039] Step S124: Calculate the indirect correlation weight of the images of the same shooting angle of non-adjacent growth stages in the initial time-series correlation network, which is determined by the sum of the products of the initial edge weights of all paths between the two nodes. The path selection rule is that the interval between the growth stages corresponding to the nodes in the path does not exceed N, and the shooting angle identification information is consistent, N being a set positive integer.

[0040] In the initial time-series correlation network, in addition to the direct initial edge weight connection between the image nodes of adjacent growth stages, there is also indirect correlation between the image nodes of the same shooting angle of non-adjacent growth stages. When calculating the indirect correlation weight of these non-adjacent nodes, all possible paths between the two nodes are considered. The path selection needs to satisfy that the interval between the growth stages corresponding to the nodes in the path does not exceed a set positive integer N, and the shooting angle identification information is consistent, to ensure the effectiveness and relevance of the path. For each path that meets the rule, the initial edge weights between all nodes on the path are multiplied to obtain the weight product of the path. Then the weight products of all paths between the two nodes are added, and the sum is the indirect correlation weight of the two non-adjacent nodes. The indirect correlation weight reflects the correlation degree established between the images of non-adjacent growth stages through intermediate growth stage images.

[0041] Step S125: Integrate the initial edge weight and the indirect correlation weight into the organ correlation coefficient between nodes to obtain the plant image time-series correlation network. The larger the organ correlation coefficient, the stronger the correlation of the corresponding organs in the two images.

[0042] The initial edge weight and the indirect association weight calculated above are integrated. For any two nodes in the network, the initial edge weight (if adjacent) and the indirect association weight between them are added to obtain an organ association coefficient between the nodes, thereby completing the construction of the plant image time sequence association network. The plant image time sequence association network can comprehensively reflect the association degree between the organs in the wheat image at different growth stages and the same shooting angle. The greater the organ association coefficient, the stronger the association of the features such as the shape and position of the corresponding organs in the two images, and the greater the continuity in the time sequence.

[0043] Step S126: Based on the organ association coefficient in the plant image time sequence association network, a reference image with an organ association coefficient higher than an association threshold value is selected, the central axis of the main stem and the main vein line of the leaf in the reference image are taken as the calibration reference, and coordinate calibration processing is performed on the corresponding organs in the current image to make the main reference lines of the same organ in different images coincide to reach the calibration standard, thereby obtaining an organ-level time sequence aligned image set.

[0044] According to the set association threshold value, an image with an organ association coefficient higher than the threshold value in the plant image time sequence association network is selected as a reference image. The reference image and the current image have strong organ association and are suitable as a calibration basis. The central axis of the main stem and the main vein line of the leaf in the reference image are taken as the calibration reference. The central axis of the main stem is a center line obtained by fitting the main stem contour point coordinates, and the main vein line of the leaf is a main vein contour line extracted by the leaf vein detection algorithm. Then, coordinate calibration processing is performed on the corresponding organs in the current image. The coordinate system of the current image is adjusted so that the central axis of the main stem in the current image coincides with the central axis of the main stem in the reference image, and the main vein line of the leaf coincides with the main vein line of the leaf in the reference image, thereby ensuring that the main reference lines of the same organ in different images coincide to reach the set calibration standard. After the above calibration processing, an organ-level time sequence aligned image set is obtained. The images in the organ-level time sequence aligned image set have high consistency in the time sequence position of the organs.

[0045] Step S130: The organ-level time sequence aligned image set is subjected to cross-growth-stage lesion feature association processing through the organ association coefficient, thereby obtaining a cross-stage lesion association set and time sequence evolution information of the lesions on different organs.

[0046] Step S131: Each image in the organ-level time sequence aligned image set is subjected to plant organ segmentation processing. A segmentation method based on a plant organ shape template is adopted to segment the image into a leaf region image, a main stem region image, and a branch region image, and the coordinate range information and organ type identifier of each organ region image are recorded.

[0047] For each wheat image in the organ-level time-aligned image set, a segmentation method based on plant organ morphological templates is adopted for processing. Morphological templates of wheat leaf, main stem and branch are constructed in advance, which contain typical morphological characteristics of each organ at different growth stages. The image is matched with these templates, and the image is segmented into leaf region image, main stem region image and branch region image according to the matching result. For example, the region with the highest matching degree with the leaf morphological template is segmented into the leaf region image, and the main stem region image and the branch region image are obtained in the same way. At the same time, the coordinate range information of each organ region image in the original image, such as the coordinates of the upper left corner and the lower right corner, and the corresponding organ type identifier, such as "leaf", "main stem" and "branch", are recorded, so as to separately process the lesion characteristics of different organs in the subsequent process.

[0048] Step S132: Perform preliminary lesion detection processing on each organ region image, adopt a sliding window detection method adapted to the shape of the plant organ, the shape of the window is determined according to the organ type, the leaf region adopts an elliptical window, the main stem region adopts a rectangular window, and the branch region adopts an elongated window, move the window by a set step to traverse the organ region image, and extract the gray scale change features and color distribution features in the window.

[0049] Step S1321: Determine the parameters of the sliding window according to the organ type identifier, the leaf region adopts an elliptical window, the length of the major axis of the elliptical window is set to a fixed proportion of the longest side of the leaf region image, and the length of the minor axis is a fixed proportion of the length of the major axis; the main stem region adopts a rectangular window, the length of the window is set to a fixed proportion of the length of the main stem region image, and the width is set to a fixed proportion of the width of the main stem region image; the branch region adopts an elongated window, the length of the window is set to a fixed proportion of the length of the branch region image, and the width is set to a fixed proportion of the width of the branch region image; the sliding step is uniformly set to a fixed proportion of the shortest side of the corresponding window.

[0050] According to the organ type identifier, select appropriate sliding window parameters for different organ region images. For leaf region images, an elliptical window is adopted, the length of the major axis of the elliptical window is set to a fixed proportion of the longest side of the leaf region image, and the length of the minor axis is a fixed proportion of the length of the major axis, which is more suitable for the natural shape of the leaf. The main stem region image adopts a rectangular window, the length of the window is a fixed proportion of the length of the main stem region image, and the width is a fixed proportion of the width of the main stem region image, which is suitable for the long strip shape of the main stem. The branch region image adopts an elongated window, the length and width of the window are a fixed proportion of the length and width of the branch region image, which is in line with the characteristics of the thin branch. The sliding step is uniformly set to a fixed proportion of the shortest side of the corresponding window, which ensures that the window can uniformly traverse the entire organ region image.

[0051] Step S1322: Moving the window by a set sliding step, traversing the whole organ region image in the order from left to right and from top to bottom, and recording the position coordinates of each window and the image pixel data in the window.

[0052] According to the set sliding step, the sliding window is moved in the order from left to right and from top to bottom to traverse the whole organ region image. During the traversal, the position coordinates of each window in the organ region image, such as the coordinates of the upper left corner of the window, and the image pixel data contained in the window are recorded. The pixel data is the basis for subsequent extraction of the gray scale change feature and the color distribution feature.

[0053] Step S1323: Extracting the gray scale change feature of the pixel data in each window, converting the pixel data into a gray scale image, calculating the gray scale value standard deviation, the gray scale value gradient mean value and the gray scale value entropy of the gray scale image in the window, and combining the gray scale value standard deviation, the gray scale value gradient mean value and the gray scale value entropy of the gray scale image in a fixed order as the gray scale change feature. The gray scale value standard deviation reflects the dispersion degree of the gray scale distribution in the window, the gray scale value gradient mean value reflects the intensity of the gray scale change in the window, and the gray scale value entropy reflects the uniformity of the gray scale distribution in the window.

[0054] The pixel data in each window is converted into a gray scale image, and the gray scale value of each pixel in the gray scale image is obtained by weighted average of the RGB value of the original pixel. Then the gray scale value standard deviation of the gray scale image is calculated. The greater the standard deviation, the greater the dispersion degree of the gray scale distribution in the window, that is, the more obvious the difference between light and dark. The gray scale value gradient mean value is calculated, and the gradient mean value reflects the intensity of the gray scale change in the window. The greater the gradient mean value, the more intense the gray scale change, and there may be edges or textures. The gray scale value entropy is calculated. The greater the entropy value, the more uneven the gray scale distribution in the window, and the more complex the information contained. The three indicators are combined in a fixed order to form the gray scale change feature of the image in the window.

[0055] Step S1324: Extracting the color distribution feature of the pixel data in each window, converting the pixel data into a hue-saturation-lightness color space, and calculating the histogram of the hue channel, the saturation channel and the lightness channel respectively; dividing each histogram into a fixed number of intervals, calculating the proportion of the number of pixels in each interval to the total number of pixels in the window, and combining the interval pixel proportions of the three channel histograms in the channel order as the color distribution feature.

[0056] The pixel data in the window is converted from the RGB color space to the hue-saturation-lightness color space, which is more consistent with the way the human eye perceives colors. Then, histograms are calculated for the hue channel, the saturation channel, and the lightness channel, respectively. The histograms can reflect the distribution of pixels with different color values in each channel. Each histogram is divided into a fixed number of intervals, for example, the hue channel histogram is divided into multiple intervals, each of which represents a range of hue values. The proportion of the number of pixels in each interval to the total number of pixels in the window is calculated, and these proportion values reflect the distribution of the color in the interval in the window. The interval pixel proportions of the histograms of the three channels are combined in the order of hue, saturation, and lightness to obtain the color distribution features.

[0057] Step S133: Collect image samples of healthy organs of the same plant species, extract the gray level variation features and color distribution features of the corresponding organ regions in the image samples, calculate the mean range and distribution law of the sample features, and determine the normal feature range of each organ.

[0058] A large number of organ image samples of healthy wheat of the same species are collected, which should cover healthy leaf, main stem and branch images at different growth stages and different shooting angles. The above healthy sample images are subjected to the same organ region segmentation and sliding window processing as described above, and the gray level variation features and color distribution features in each window are extracted. Then, the mean range of these sample features is calculated, i.e. the average value of each feature index and the upper and lower fluctuation range, and the distribution law is analyzed, such as whether it conforms to the normal distribution or not. According to the mean range and distribution law of the sample features, the normal feature range of each organ at different growth stages and different positions is determined. When the subsequently detected window features exceed the normal range, the window region may have a lesion.

[0059] Step S134: Select the window region whose gray level variation features or color distribution features deviate from the normal feature range of the corresponding organ as a suspected lesion region, record the position coordinates, size information and corresponding feature data of each suspected lesion region, form an organ suspected lesion information table for each image, and the feature data includes color distribution histogram data, texture distribution feature data and shape parameter data.

[0060] The gray scale variation features and color distribution features of each window extracted above are compared with the normal feature range of the corresponding organ. If any index in the gray scale variation features (such as the standard deviation of the gray scale value, the average gradient, or the entropy) of a window deviates from the normal range, or the interval pixel ratio in the color distribution features deviates from the normal range, the window region is marked as a suspected lesion region. The position coordinates of each suspected lesion region in the organ region image, the size information (such as the length and width of the window), and the corresponding feature data are recorded, including the color distribution histogram data (the interval pixel ratio of each channel), the texture distribution feature data (the three index values of the gray scale variation features), and the shape parameter data (such as the shape, perimeter, and area of the window). The above information is arranged into an organ suspected lesion information table for each image to facilitate subsequent correlation analysis of the suspected lesion regions.

[0061] Step S135: According to the organ correlation coefficient, the correlation images with an organ correlation coefficient higher than the correlation threshold are screened out, and the organ suspected lesion information table corresponding to the same organ type identifier in the correlation images is extracted as the correlation lesion information.

[0062] According to the organ correlation coefficient calculated in the plant image time sequence correlation network, the images with an organ correlation coefficient higher than the correlation threshold are screened out as the correlation images. The above correlation images have a strong organ correlation with the current image and may contain information related to the suspected lesion regions in the current image. Then, the organ suspected lesion information table with the same organ type identifier (such as both being leaves, stems, or branches) is extracted from the correlation images as the correlation lesion information. By comparing the suspected lesion information of the current image and the correlation image, the changes of the lesion in different growth stages can be found.

[0063] Step S136: The position correspondence relationship and feature data similarity of the suspected lesion regions of the current image and the correlation image are compared. The position correspondence relationship is determined by the coordinate overlap ratio of the two suspected lesion regions in the organ coordinate system. The feature data similarity is comprehensively calculated by the coincidence degree of the color distribution histogram, the matching ratio of the texture distribution features, and the standardized value of the deviation degree of the shape parameters. The deviation degree of the shape parameters is converted into a deviation score comparable to the coincidence degree and the matching ratio. The higher the color distribution histogram coincidence degree, the higher the texture distribution feature matching ratio, and the higher the shape parameter deviation score, the higher the feature data similarity.

[0064] An organ coordinate system is established, taking the upper left corner of the organ region image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis. For each suspected lesion region of the current image, find the suspected lesion region in the organ suspected lesion information table corresponding to the same organ type identifier of the associated image, calculate the coordinate overlap ratio of the suspected lesion regions in the organ coordinate system, that is, the proportion of the area of the overlapping part of the two suspected lesion regions to the total area of the two regions, which reflects the positional correspondence. The higher the proportion, the closer the positional correspondence. At the same time, the feature data similarity is calculated, including the coincidence degree of the color distribution histogram, that is, the closeness of the proportion of the corresponding interval pixels in the color distribution histograms of the two suspected lesion regions, the higher the coincidence degree, the more similar the color distribution; the matching proportion of texture distribution features, that is, the proportion of the matching number of each index in the total number of indexes in the gray scale change feature; the deviation degree of shape parameters, which converts the difference value of the shape parameters into a deviation score in a similar manner to the branch connection angle deviation value, so that the deviation score is comparable to the coincidence degree and the matching proportion. After standardizing the three indicators, the feature data similarity is calculated. The higher the color distribution histogram coincidence degree, the higher the texture distribution feature matching proportion, and the higher the shape parameter deviation score, the higher the feature data similarity.

[0065] Step S137: Dividing the suspected lesion regions whose positional correspondence and feature data similarity both satisfy the association conditions into the same lesion association group, and each lesion association group contains associated suspected lesion regions on the same organ at different growth stages.

[0066] The association condition of the positional correspondence is that the coordinate overlap ratio is higher than a certain threshold, and the association condition of the feature data similarity is that the similarity value is higher than a certain threshold. For the suspected lesion regions in the current image and the associated image that satisfy the two association conditions, they are divided into the same lesion association group. Each lesion association group contains associated suspected lesion regions on the same organ at different growth stages, which are likely to be the same lesion at different growth stages.

[0067] Step S138: Extracting the positional coordinate change trajectory, size size change trend, and feature data change rule of each suspected lesion region in each lesion association group, generating the time sequence evolution information of the lesion on different organs, and the time sequence evolution information contains the positional offset trajectory of the lesion in the organ coordinate system, the size expansion or contraction curve, and the feature data change matrix.

[0068] For each lesion association group, arrange the suspected lesion regions in it in the order of growth stages. Extract the position coordinates of each suspected lesion region in the organ coordinate system, connect the above coordinates in time order to form a position offset trajectory, reflecting the movement of the lesion on the organ. Record the size of each suspected lesion region (such as area), analyze its expansion or contraction trend with the growth stage, and form a size expansion or contraction curve. At the same time, sort out the feature data of each suspected lesion region, such as color distribution histogram data and texture distribution feature data, analyze the change law of these data over time, construct a feature data change matrix, the rows of the matrix represent different growth stages, the columns represent different feature indicators, and the matrix elements are the values of the corresponding feature indicators. Integrate the position offset trajectory, the size expansion or contraction curve, and the feature data change matrix to generate the time evolution information of the lesion on different organs.

[0069] Step S139: Combine all lesion association groups and the suspected lesion regions contained in each group into a cross-stage lesion association set, and record the time evolution information of the lesion in each group on different organs and the corresponding organ type identifier.

[0070] Summarize all the above obtained lesion association groups, each of which contains associated suspected lesion regions on the same organ at different growth stages. Combine the above lesion association groups together to form a cross-stage lesion association set. At the same time, record the time evolution information of the corresponding lesion on different organs and the organ type identifier with the lesion association groups in the set, so that the source and evolution of each lesion association group can be clearly determined when inputting the deep learning model for subsequent processing.

[0071] Step S140: Input the cross-stage lesion association set into a pre-set plant pest and disease specific deep learning model for feature aggregation and type matching processing to obtain lesion aggregation features and preliminary pest and disease type results.

[0072] Step S141: Preprocess each lesion association group in the cross-stage lesion association set, extract the image data of each suspected lesion region in each group, and adjust the image size according to the organ type of the lesion. Leaf lesion images are adjusted to a uniform rectangular size, and main stem and branch lesion images are adjusted to a uniform elongated size.

[0073] For each lesion association group in the cross-stage lesion association set, the original image data of each suspected lesion region is extracted. According to the organ type where the lesion is located, the image data is resized. The leaf lesion image is adjusted to a uniform rectangular size to keep the length and width consistent. The stem and branch lesion image is adjusted to a uniform elongated size due to its morphological characteristics, ensuring that the image data of different lesion association groups has a uniform specification, facilitating subsequent model processing and feature extraction.

[0074] Step S142: Perform pixel value normalization processing on the adjusted image data to convert the pixel values to a fixed numerical range, obtaining a standardized lesion image sequence.

[0075] The pixel values of the resized image data may be in different ranges. In order to eliminate the influence of pixel value difference on model training and recognition, pixel value normalization processing is performed on the image data. The pixel value of each pixel is converted to a fixed numerical range, for example, from 0-255 to 0-1. Normalization processing is achieved by dividing the pixel value by the maximum value of the channel (e.g. 255) to obtain a standardized lesion image sequence. The image pixel values in the standardized lesion image sequence have a uniform range, which facilitates the model to better learn the lesion features.

[0076] Step S143: Extract the time sequence arrangement information of the standardized lesion image sequence, combine the growth stage interval data in the time sequence evolution information of the lesion on different organs, and perform time sequence completion processing on the standardized lesion image sequence. When the image interval between adjacent growth stages exceeds the interval threshold, an interpolation method based on the evolution law of organ lesions is used to generate intermediate images, obtaining a complete time sequence lesion image sequence.

[0077] From the standardized lesion image sequence, the time sequence arrangement information, i.e. the growth stage order corresponding to each image, is extracted. Combined with the growth stage interval data in the time sequence evolution information of the lesion on different organs, it is checked whether the image interval between adjacent growth stages exceeds the set interval threshold. If the interval exceeds the threshold, it means that there is a missing in the time sequence, which needs to be completed. The completion method uses an interpolation method based on the evolution law of organ lesions, which predicts the lesion image features of the missing growth stage according to the known lesion image features and time sequence evolution information before and after, and generates intermediate images. For example, according to the position offset trajectory, size change trend and feature data change law of the lesion, the possible morphology and features of the lesion in the intermediate growth stage are inferred, and the corresponding image is generated. Through the above time sequence completion processing, a complete time sequence lesion image sequence is obtained, ensuring the continuity and integrity of the time sequence information.

[0078] Step S144: input the complete time sequence lesion image sequence into the feature input layer of the plant pest disease specific deep learning model, and through the dimension conversion operation of the input layer, convert the image sequence into an input format suitable for each branch of the model, to obtain branch input features.

[0079] The complete time sequence lesion image sequence is input into the feature input layer of the preset plant pest disease specific deep learning model. The feature input layer first performs a dimension conversion operation on the input image sequence, adjusts the dimensions of the image sequence to an input format acceptable by each subsequent parallel branch of the model. For example, the channel number, height, width and other dimensions of the image sequence are adjusted to match the input requirements of the lesion texture detail extraction branch, the lesion morphological feature extraction branch and the lesion color feature extraction branch. After dimension conversion, branch input features are obtained, which can be correctly received and processed by each branch.

[0080] Step S145: input the branch input features into the lesion texture detail extraction branch, and through the sequential processing of the preset convolutional layer, pooling layer and batch normalization layer, extract the lesion surface texture details in the image to obtain a lesion texture feature map.

[0081] The branch input features are first input into the lesion texture detail extraction branch. The lesion texture detail extraction branch includes multiple convolutional layers, multiple pooling layers and two batch normalization layers. The convolutional layer performs convolutional operations on the input features through convolutional kernels of different sizes to extract local features in the image. As the convolutional layers deepen, the extracted features gradually transition from low-level edges and textures to high-level semantic features. The pooling layer uses a max-pooling method to downsample the feature map output by the convolutional layer, reducing the size and number of parameters of the feature map while preserving important feature information. The batch normalization layer is inserted between two adjacent convolutional layers to standardize the input of each layer, accelerating the training convergence of the network and improving the generalization ability of the model. Through the sequential processing of the above layers, the texture details of the lesion surface in the image, such as the texture of spots and the hole texture of insect damage, can be effectively extracted to obtain a lesion texture feature map.

[0082] Step S146: synchronously input the branch input features into the lesion morphological feature extraction branch, and through the expanded receptive field processing of the atrous convolutional layer and the global information aggregation processing of the global average pooling layer, extract the overall morphological features of the lesion to obtain a lesion morphological feature map.

[0083] The branch input feature is also input into the lesion color feature extraction branch. The lesion color feature extraction branch includes multiple convolution layers, multiple fully connected layers, and a random dropout layer. The convolution layer is used to extract the color feature in the image, and the distribution and change information of the color is captured through different convolution kernels. The fully connected layer converts and integrates the color feature extracted by the convolution layer, and maps the high-dimensional feature to a low-dimensional space. The number of neurons of the fully connected layer decreases successively, and the feature dimension is compressed gradually. The random dropout layer is inserted between two adjacent fully connected layers to prevent model overfitting by randomly discarding part of the neurons with a certain probability. Through the above processing, the color distribution feature of the lesion, such as the composition ratio of the color, the change of the color depth, and the color gradient feature, such as the color transition from the center to the edge of the lesion, can be extracted, and finally the lesion color feature vector is obtained.

[0084] Step S147: The branch input feature is input into the lesion color feature extraction branch synchronously. Through the color feature extraction of the convolution layer and the feature conversion processing of the fully connected layer, the color distribution feature and the color gradient feature of the lesion are extracted, and the lesion color feature vector is obtained.

[0085] The branch input feature is also input into the lesion color feature extraction branch. The lesion color feature extraction branch includes multiple convolution layers, multiple fully connected layers, and a random dropout layer. The convolution layer is used to extract the color feature in the image, and the distribution and change information of the color is captured through different convolution kernels. The fully connected layer converts and integrates the color feature extracted by the convolution layer, and maps the high-dimensional feature to a low-dimensional space. The number of neurons of the fully connected layer decreases successively, and the feature dimension is compressed gradually. The random dropout layer is inserted between two adjacent fully connected layers to prevent model overfitting by randomly discarding part of the neurons with a certain probability. Through the above processing, the color distribution feature of the lesion, such as the composition ratio of the color, the change of the color depth, and the color gradient feature, such as the color transition from the center to the edge of the lesion, can be extracted, and finally the lesion color feature vector is obtained.

[0086] Step S148: The lesion texture feature map, the lesion morphology feature map, and the lesion color feature vector are input into the lesion feature interaction layer. The lesion texture feature map and the lesion morphology feature map are spliced according to the channel dimension, and then a spatial weight map is generated through a spatial attention mechanism. The spliced feature map is processed through the spatial weight map, and a channel weight vector is generated through a channel attention mechanism. The channel weight vector is used to process the feature map after the spatial weighting through the channel weighting, and an interaction feature map is obtained.

[0087] The lesion texture feature map, the lesion shape feature map and the lesion color feature vector output by the three branches are input into the lesion feature interaction layer. First, the lesion texture feature map and the lesion shape feature map are spliced in the channel dimension, that is, the two feature maps are combined together in the depth direction to form a higher-dimensional spliced feature map. Then, the spliced feature map is processed through a spatial attention mechanism, which can focus on the spatial regions more important for pest and disease identification in the feature map to generate a spatial weight map. Each element in the spatial weight map represents the weight value of the corresponding position, and the higher the weight value, the more important the feature of the position. The spatial weight map is used for spatial weighting processing of the spliced feature map, that is, element-by-element multiplication, to enhance the features of important spatial regions and suppress the features of unimportant regions. Then, the spatially weighted feature map is processed through a channel attention mechanism, which focuses on the importance of different channel features to generate a channel weight vector. Each element in the channel weight vector represents the weight value of the corresponding channel, and the vector is used for channel weighting processing of the spatially weighted feature map, that is, the feature map of each channel is multiplied by the corresponding weight value to highlight the features of important channels. After the double weighting processing of space and channel, an interaction feature map is obtained, which integrates texture, shape and color features and highlights key information.

[0088] Step S149: input the interaction feature map into the lesion feature aggregation layer to obtain a lesion aggregation feature vector of fixed length through deepening extraction by a plant lesion exclusive convolutional neural network block and dimension unification processing by an adaptive pooling layer.

[0089] The interaction feature map is input into the lesion feature aggregation layer. The lesion feature aggregation layer first inputs the interaction feature map into a plant lesion exclusive convolutional neural network block, which includes multiple convolution operations. The convolution operations further deepen the extraction of comprehensive features of the lesion and integrate different dimensional lesion features, such as more closely integrating the texture, shape and color features after attention weighting, and optimizing feature expression to make the features more discriminative. Then, the feature map output by the convolutional neural network block is processed through an adaptive pooling layer. The adaptive pooling layer can automatically adjust the size of the pooling region according to the preset output size to convert feature maps of different sizes into a feature vector of fixed length. Thus, a lesion aggregation feature vector of fixed length is obtained, which integrates various feature information of the lesion and is suitable for input into the subsequent pest and disease type matching layer for classification.

[0090] Step S1410: input the lesion aggregation feature into the pest and disease type matching layer to obtain probability values corresponding to each pest and disease type through feature mapping by two fully connected layers and probability conversion by a probability activation function, and select the pest and disease type with the highest probability value as the preliminary pest and disease type result of the lesion association group.

[0091] The lesion aggregation feature vector is input into a pest disease type matching layer. The pest disease type matching layer includes two fully connected layers and a probability activation function. The first fully connected layer maps the lesion aggregation feature vector to an intermediate dimension feature vector, further integrating and converting the features. The second fully connected layer maps the intermediate dimension feature vector to a dimension consistent with the number of known plant pest disease types, with each dimension corresponding to a pest disease type. Then, the output of the second fully connected layer is converted into probability values corresponding to each pest disease type by a probability activation function (such as a softmax function), and the sum of the probability values is 1. The pest disease type with the highest probability value is selected as the preliminary pest disease type result of the lesion association group, and it is preliminarily determined that the lesion may belong to which pest disease type.

[0092] Step S150: Perform organ adaptation verification processing on the preliminary pest disease type result in combination with the time evolution information of the lesions on different organs to generate a final plant pest disease identification result and a lesion organ time evolution map.

[0093] Step S151: Extract the time evolution information of the lesions on different organs of each lesion association group in the cross-stage lesion association set, and analyze the position offset trajectory, size expansion or contraction curve, and feature data change matrix of the suspected lesion area in the organ coordinate system in each growth stage.

[0094] From the cross-stage lesion association set, the time evolution information of the lesions on different organs corresponding to each lesion association group is extracted. The time evolution information is analyzed to separate the position offset trajectory of the suspected lesion area in the organ coordinate system in each growth stage, which is composed of a series of coordinate points and reflects the movement path of the lesion on the organ; the size expansion or contraction curve, which takes the growth stage as the horizontal axis and the lesion size (such as area) as the vertical axis, and displays the trend of the lesion size change over time; and the feature data change matrix, which records the numerical changes of various feature data (such as gray change features, color distribution features, etc.) of the lesions in different growth stages.

[0095] Step S152: Obtain a preset plant pest disease organ time evolution feature library, which includes standard time evolution rules of various known plant pest disease types on different organs. Each pest disease type corresponds to a set of standard evolution rules, including standard position offset trajectory on leaves, standard size change curve, and standard feature data change rule, and the standard evolution rules on the main stem and branches are the same.

[0096] The preset plant pest organ time evolution feature library is established by analyzing and summarizing a large number of known plant pest types. The plant pest organ time evolution feature library contains standard time evolution rules of a plurality of known plant pest types on different organs (leaves, main stems, branches). For each pest type, there is a standard position offset trajectory on the leaves, that is, the pest usually starts from which position of the leaves and how to spread and move; a standard size change curve, that is, the typical expansion or contraction pattern of the lesion size with the growth stage; and a standard feature data change rule, that is, the standard change of the gray scale change feature, the color distribution feature, and the like with time. There are also similar standard evolution rules on the main stems and branches.

[0097] Step S153: According to the organ type identifier corresponding to the lesion correlation group, the standard time evolution rules of each pest type under the organ type are matched from the plant pest organ time evolution feature library. The position offset trajectory of the lesion correlation group is compared with the matched standard position offset trajectory, and the trajectory matching degree is calculated.

[0098] According to the organ type identifier corresponding to the lesion correlation group, such as leaves, main stems, or branches, the standard time evolution rules of all known pest types under the organ type are matched from the plant pest organ time evolution feature library. Then, the position offset trajectory of the current lesion correlation group is compared with the standard position offset trajectory of each pest type matched. When comparing, the distance between the coordinate points of the two trajectories at the corresponding growth stages is calculated, and the average of the above distance is taken as the measure of the trajectory difference. Then, the trajectory difference is converted into the trajectory matching degree. The smaller the difference is, the higher the matching degree is. The trajectory matching degree reflects the similarity between the position change of the current lesion and the position change of the standard pest.

[0099] Step S154: The size expansion or contraction curve is compared with the standard size change curve to calculate the size matching degree; and the feature data change matrix is compared with the standard feature data change rule to calculate the feature matching degree.

[0100] Similarly, the size expansion or contraction curve of the lesion correlation group is compared with the standard size change curve. The difference between the size values of the two curves at the corresponding growth stages is calculated, and the average of the difference is converted into the size matching degree. The smaller the difference is, the higher the size matching degree is. For the feature data change matrix, it is compared with the standard feature data change rule. For each feature index in the matrix, the similarity between the change trend of the index with the growth stage and the change trend of the corresponding index in the standard rule is calculated, such as by calculating the correlation coefficient to measure the similarity. Then, the similarities of all the feature indexes are weighted and averaged to obtain the feature matching degree. The feature matching degree reflects the similarity between the feature change of the current lesion and the feature change of the standard pest.

[0101] Step S155: Assign organ adaptation weights to the trajectory matching degree, size matching degree and feature matching degree, and calculate the comprehensive temporal matching degree by weighted summation. The trajectory matching degree weight of the leaf lesion is the highest, the size matching degree weight of the main stem lesion is the highest, and the feature matching degree weight of the branch lesion is the highest.

[0102] According to the characteristics of different organs, different organ adaptation weights are assigned to the trajectory matching degree, size matching degree and feature matching degree. For leaf lesions, the position of insect pests on the leaf moves relatively obviously, and the importance of trajectory matching degree for recognition is higher, so the trajectory matching degree weight is the highest. For main stem lesions, the size of the lesion usually changes significantly, such as the stem decay area caused by stem boring pests, which will gradually expand, so the size matching degree weight is the highest. For branch lesions, the change of feature data (such as color and texture) may be more unique, and the feature matching degree weight is the highest. Then, the trajectory matching degree, size matching degree and feature matching degree are multiplied by their respective weights, and the products are added to obtain the comprehensive temporal matching degree. The comprehensive temporal matching degree comprehensively considers the matching degree of the lesion in position, size and feature with the standard insect pest.

[0103] Step S156: If the comprehensive temporal matching degree is higher than the organ adaptation verification threshold, it is confirmed that the preliminary insect pest type result of the lesion associated group is the effective insect pest type result.

[0104] An organ adaptation verification threshold is set, which is determined according to a large amount of experimental data and experience. The calculated comprehensive temporal matching degree is compared with the threshold. If the comprehensive temporal matching degree is higher than the threshold, it means that the temporal evolution characteristics of the current lesion associated group are highly matched with the standard temporal evolution rule corresponding to the preliminary insect pest type result, and the preliminary result is reliable, so the preliminary insect pest type result is confirmed as the effective insect pest type result.

[0105] Step S157: If the comprehensive temporal matching degree is not higher than the organ adaptation verification threshold, the lesion aggregation features of the lesion associated group and the corresponding organ type identifier and growth stage information are input into the dynamic optimization module, triggering the model parameter fine-tuning to re-perform type matching to obtain a secondary recognition type result, and then repeat the organ adaptation verification step until the effective insect pest type result is obtained.

[0106] If the comprehensive temporal matching degree is not higher than the organ adaptation verification threshold, it indicates that the preliminary insect disease type result may be inaccurate, and further processing is needed. The lesion aggregation features of the lesion correlation group and the corresponding organ type identifier and growth stage information are input into the dynamic optimization module of the model. After receiving this information, the dynamic optimization module triggers model parameter fine-tuning. The fine-tuning process uses these samples with uncertainties to retrain the model in small batches, adjusts the weight parameters of each layer of the model, and improves the model's recognition ability for this type of lesion. After fine-tuning, the complete temporal lesion image sequence of the lesion correlation group is re-input into the optimized model for type matching to obtain a secondary recognition type result. Then, repeat the organ adaptation verification process of steps S151 to S156, calculate the comprehensive temporal matching degree of the secondary recognition type result, and compare it with the threshold. If it still does not meet the requirement, continue to trigger model parameter fine-tuning, repeat the matching and verification steps, and obtain an effective insect disease type result with a comprehensive temporal matching degree higher than the threshold.

[0107] Step S158: Collect all the effective insect disease type results of the lesion correlation group, associate the corresponding growth stage time identifier, organ type identifier, organ region coordinate range information, and feature data of the suspected lesion area, and construct a prediction association table, which contains the unique identifier of each record.

[0108] Collect all the effective insect disease type results of the lesion correlation group after organ adaptation verification. Associate each effective insect disease type result with the corresponding growth stage time identifier (such as the heading stage of a certain month in a certain year), organ type identifier (leaf, main stem, or branch), organ region coordinate range information (coordinates of the organ in the original image), and feature data of the suspected lesion area (color distribution histogram data, texture distribution feature data, shape parameter data, etc.). Assign a unique identifier, such as a unique string or number, to each association record to distinguish different lesion correlation groups. Organize the above association information into a prediction association table, which clearly shows the detailed information and recognition results of each lesion correlation group.

[0109] Step S159: Construct a graph coordinate system with growth stage as the horizontal axis and plant organ type as the vertical axis. The horizontal axis is labeled in the order of growth stages, and the vertical axis is labeled in the order of leaves, main stems, and branches. Draw the outline identifier of each organ in the graph coordinate system according to the organ region coordinate range information. The shape of the outline identifier is consistent with the actual shape of the corresponding organ.

[0110] A coordinate system of a lesion organ time evolution graph is constructed, taking the growth stage as the horizontal axis and marking it in the order of seed germination stage, tillering stage, jointing stage, heading stage, and grain filling stage; taking the plant organ type as the vertical axis and marking it in the order of leaf, main stem, and branch. According to the coordinate range information of each organ area recorded in the prediction association table, the outline mark of each organ is drawn in the graph coordinate system. For example, the outline of the leaf is drawn according to the coordinate range of the leaf area at the vertical axis position corresponding to the leaf and the horizontal axis position corresponding to the growth stage, and the shape of the outline is consistent with the actual morphology of the wheat leaf at the growth stage; the outline marks of the main stem and the branch are drawn in the same way.

[0111] Step S1510: In each organ outline mark, a lesion mark is drawn according to the position coordinates of the suspected lesion area, the color of the lesion mark corresponds to the effective pest disease type result one by one, the size of the lesion mark is proportional to the size of the suspected lesion area, and the unique identifier of the lesion association group is marked in the lesion mark.

[0112] In each organ outline mark, the specific position of the lesion in the outline mark is determined according to the position coordinates of the suspected lesion area in the organ coordinate system, and the lesion mark is drawn. The color of the lesion mark corresponds to the effective pest disease type result one by one, for example, the lesion mark corresponding to the wheat rust is set to red, the lesion mark corresponding to the wheat powdery mildew is set to white, etc., and different pest disease types are intuitively distinguished by color. The size of the lesion mark is proportional to the size of the suspected lesion area, the larger the size, the larger the mark, reflecting the severity of the lesion. The unique identifier of the lesion association group is marked in each lesion mark, which facilitates the query of detailed information of the lesion through the prediction association table.

[0113] Step S1511: In the graph, connect the lesion marks of the same lesion association group at different growth stages with a polyline, the thickness of the polyline is proportional to the organ association coefficient of the lesion association group, and the key nodes of characteristic data changes are marked beside the polyline.

[0114] For the same lesion association group, connect the lesion marks at different growth stages with a polyline in the graph according to the order of the growth stages. The thickness of the polyline is proportional to the organ association coefficient of the lesion association group, the larger the organ association coefficient, the stronger the correlation of the lesions at different growth stages, and the thicker the polyline. The key nodes of characteristic data changes are marked beside the polyline, such as the growth stage at which the lesion size starts to expand rapidly, the growth stage at which the color feature changes significantly, etc. These key nodes reflect the important changes in the evolution process of the lesion.

[0115] Step S1512: Add a legend on one side of the atlas, mark the corresponding colors of different effective pest disease types, the size range of lesion identification size, the organ correlation coefficient range corresponding to the line thickness, and the meaning of the unique identification, combine the prediction correlation table with the completed atlas to form the final plant pest disease identification result.

[0116] Add a legend on one side of the atlas, and the legend details the corresponding colors of different effective pest disease types, such as red for wheat rust and white for powdery mildew; mark the size range corresponding to the lesion identification size, such as the size range corresponding to large identification is how much, the size range corresponding to medium identification is how much, etc.; mark the organ correlation coefficient range corresponding to the line thickness, such as the correlation coefficient range corresponding to thick line is how much, the correlation coefficient range corresponding to medium thickness is how much, etc.; and explain the meaning of the unique identification, such as what information does the unique identification consist of and represent respectively. Combining the prediction correlation table with the completed lesion organ time evolution atlas forms the final plant pest disease identification result. The plant pest disease identification result not only gives the type of pest disease, but also shows the evolution process of the lesion in different organs and different growth stages.

[0117] Step S210: Construction and optimization process of the preset plant pest disease specific deep learning model.

[0118] Step S211: Construct the model infrastructure, which includes a feature input layer, parallel plant lesion feature extraction branches, a lesion feature interaction layer, a lesion feature aggregation layer, a pest disease type matching layer, and a dynamic optimization module. The parallel plant lesion feature extraction branches are lesion texture detail extraction branches, lesion morphological feature extraction branches, and lesion color feature extraction branches.

[0119] The plant pest disease specific deep learning model infrastructure is constructed, which consists of multiple functional layers and modules. First is the feature input layer, which is responsible for receiving input image data and performing preliminary dimension conversion. Then there are parallel plant lesion feature extraction branches, which include three independent branches: lesion texture detail extraction branches, lesion morphological feature extraction branches, and lesion color feature extraction branches, which extract lesion features from different angles. Next is the lesion feature interaction layer, which is used to fuse the features extracted by the three branches and highlight key information. The lesion feature aggregation layer further extracts and unifies the dimensions of the interacted features. The pest disease type matching layer classifies and outputs the probability of pest disease types based on the aggregated features. The dynamic optimization module is used to trigger model parameter fine-tuning when the model identification result is uncertain, improving the identification accuracy. The above layers and modules cooperate with each other to form a complete model infrastructure.

[0120] Step S212: configuring network parameters of the lesion texture detail extraction branch, the lesion texture detail extraction branch comprising a plurality of convolutional layers, a plurality of pooling layers, and two batch normalization layers, convolution kernel sizes of the plurality of convolutional layers being sequentially increased, a step length being kept as a fixed value, the pooling layers adopting a maximum pooling manner, a pooling kernel size being set as a fixed value, and the batch normalization layers being inserted between adjacent two convolutional layers.

[0121] The network parameters of the lesion texture detail extraction branch are configured. The lesion texture detail extraction branch comprises a plurality of convolutional layers, convolution kernel sizes of which are sequentially increased from a small value as the number of layers increases, so that more complex texture features can be gradually extracted. The step length of all convolutional layers is kept as a fixed value to ensure that the size of the feature map is reduced in a certain rule. The pooling layers adopt a maximum pooling manner, which can retain the significant features of the texture, and the pooling kernel size is set as a fixed value, such as 2x2. The batch normalization layers are inserted between adjacent two convolutional layers, and two batch normalization layers are configured, which are respectively located between the early convolutional layers to accelerate network convergence and prevent overfitting. Through the above parameter configuration, the lesion texture detail extraction branch can effectively extract the surface texture detail features of the lesion.

[0122] Step S213: configuring network parameters of the lesion morphology feature extraction branch, the lesion morphology feature extraction branch comprising a plurality of dilated convolutional layers, a global average pooling layer, and a batch normalization layer, a dilated rate of the dilated convolutional layers being sequentially increased by a fixed proportion, a pooling kernel size of the global average pooling layer being set as a fixed value, and the batch normalization layer being inserted after the middle dilated convolutional layer.

[0123] The network parameters of the lesion morphology feature extraction branch are configured. The lesion morphology feature extraction branch uses a plurality of dilated convolutional layers, the dilated rate of which is sequentially increased by a fixed proportion, the increase of the dilated rate expands the receptive field, and can capture larger range of lesion morphology information. A global average pooling layer is set, and the pooling kernel size is set as a fixed value, such as the same as the feature map size output by the last dilated convolutional layer, which is used for global information aggregation of the entire feature map to obtain global morphology features. A batch normalization layer is inserted after the middle dilated convolutional layer to standardize the output of the dilated convolution, stabilize the training process, and the above parameter configuration enables the branch to effectively extract the overall morphology features of the lesion.

[0124] Step S214: configuring network parameters of the lesion color feature extraction branch, the lesion color feature extraction branch comprising a plurality of convolutional layers, a plurality of fully connected layers, and a random dropout layer, a number of neurons of the fully connected layers being sequentially decreased, a dropout probability of the random dropout layer being set as a fixed value, and the random dropout layer being inserted between adjacent two fully connected layers.

[0125] The network parameters of the lesion color feature extraction branch are configured. The lesion color feature extraction branch includes multiple convolutional layers for preliminary extraction of color features. Subsequently, multiple fully connected layers are connected, and the number of neurons of the fully connected layers decreases from more to less in turn, gradually compressing the feature dimension. A random dropout layer is inserted between adjacent two fully connected layers, and the dropout probability of the random dropout layer is set to a fixed value, such as 0.5, to prevent overfitting of the fully connected layer. Through the above parameter configuration, the lesion color feature extraction branch configured in this way can extract and integrate the color distribution and gradient features of the lesion.

[0126] Step S215: Configure the feature interaction strategy of the lesion feature interaction layer. The feature interaction strategy is to splice the feature maps output by the parallel plant lesion feature extraction branches in the channel dimension, generate a spatial weight map through a spatial attention mechanism, perform spatial weighting processing on the spliced feature map through the spatial weight map, generate a channel weight vector through a channel attention mechanism, and perform channel weighting processing on the spatially weighted feature map through the channel weight vector to obtain an interaction feature map.

[0127] The feature interaction strategy of the lesion feature interaction layer is configured. First, the feature maps output by the three parallel branches are spliced in the channel dimension to form a spliced feature map. Then, the spatial attention mechanism is applied to generate a spatial weight map by performing global average pooling and fully connected layer processing on the spliced feature map. The spatial weight map is used to perform spatial weighting on the spliced feature map to enhance the features of important spatial regions. Next, the channel attention mechanism is applied to generate a channel weight vector by performing global average pooling and fully connected layer processing on the spatially weighted feature map. The channel weight vector is used to perform channel weighting on the feature map to highlight the features of important channels. Through the above double weighting interaction strategy of space and channel, an interaction feature map that integrates key information is obtained.

[0128] Step S216: Configure the aggregation mode of the lesion feature aggregation layer. The aggregation mode is to input the interaction feature map output by the feature interaction layer into a convolutional neural network block, which includes multiple convolutional operations for deep extraction of lesion comprehensive features, fusion of different dimensional lesion features, and optimization of feature expression, and then convert the feature map into a fixed-length lesion aggregation feature vector through an adaptive pooling layer.

[0129] The aggregation manner of the lesion feature aggregation layer is configured. The interaction feature map is input into a plant lesion specific convolutional neural network block, which includes multiple layers of convolutional operations, and the size and number of convolutional kernels are set according to the feature extraction requirements. The convolutional operations further deepen the extraction of comprehensive features of the lesion, fuse the lesion features from different dimensions (texture, morphology, color), and optimize the feature expression, so that the features are more discriminative. Then, the feature map output by the convolutional neural network block is processed through an adaptive pooling layer. The adaptive pooling layer can automatically adjust the pooling operation according to the preset output length, convert the feature maps of different sizes into fixed-length lesion aggregation feature vectors, and facilitate subsequent type matching.

[0130] Step S217: The output strategy of the pest disease type matching layer is configured. The pest disease type matching layer includes two fully connected layers and a probability activation function. The first fully connected layer maps the fixed-length lesion aggregation feature vector to an intermediate-dimensional feature vector, and the second fully connected layer maps it to a dimension consistent with the number of known plant pest disease types. The probability activation function converts the output into probability values corresponding to each pest disease type.

[0131] The output strategy of the pest disease type matching layer is configured. The pest disease type matching layer includes two fully connected layers. The first fully connected layer receives the fixed-length lesion aggregation feature vector and maps it to an intermediate-dimensional feature vector. The size of the intermediate dimension is determined according to the number of pest disease types and the complexity of the features. The second fully connected layer maps the intermediate-dimensional feature vector to a dimension consistent with the number of known plant pest disease types, with each dimension corresponding to a pest disease type. Finally, a probability activation function (such as a softmax function) is used to process the output of the second fully connected layer, converting it into probability values corresponding to each pest disease type, which represent the confidence of the model for that pest disease type.

[0132] Step S218: The optimization rules of the dynamic optimization module are configured. The optimization rules are the deviation data between the pest disease type probability values output by the model in real time and the pest disease types labeled by the samples. When the deviation values of a continuous fixed number of samples are higher than the deviation threshold, the model parameter fine-tuning is triggered.

[0133] The optimization rules of the dynamic optimization module are configured. The dynamic optimization module collects deviation data between the model output pest disease type probability value and the sample labeled true pest disease type in real time. The deviation data can be measured by calculating the cross-entropy loss between the predicted probability distribution and the true label distribution. When the deviation values of a continuous fixed number of samples are all higher than the set deviation threshold, it indicates that the model performance on the current identification task has decreased or there are insufficiently learned sample patterns, and at this time, the model parameter fine-tuning is triggered. When fine-tuning, a small batch of data sets composed of these high deviation samples are used to perform several rounds of iterative training on the model to adjust the weight parameters of each layer to reduce the deviation and improve the model performance.

[0134] Step S219: Collecting a labeled sample set containing different plant pest disease types, the labeled sample set containing lesion images of each pest disease type in different plant organs and different growth stages, each sample labeled with pest disease type name, lesion organ type, lesion growth stage, and lesion severity level information.

[0135] A large number of labeled sample sets containing different plant pest disease types are collected. The sample set should cover a variety of common plant pest disease types, such as rust, powdery mildew, and aphid damage of wheat. For each pest disease type, collect lesion images in different plant organs (leaves, main stems, branches), different growth stages (seed germination to grain filling). Each sample is labeled in detail, including pest disease type name, lesion organ type, lesion growth stage, and lesion severity level (such as mild, moderate, severe) information.

[0136] Step S2110: Divide the labeled sample set into training set, validation set and test set according to fixed proportion, the training set contains samples of different pest disease types, different organ types and growth stages, the sample distribution of validation set and test set keeps consistent with training set.

[0137] The labeled sample set is divided into training set, validation set and test set according to fixed proportion, such as 7:2:1 ratio. During the division process, ensure that the training set contains all different pest disease types, different organ types and different growth stages of samples, and the proportion of each type of sample is as balanced as possible. The sample distribution of the validation set and the test set keeps consistent with the training set, that is, the proportion of pest disease types, organ types and growth stages is the same as the training set, to ensure the reliability of the validation and test results. The training set is used for parameter learning of the model, the validation set is used for model performance evaluation and parameter adjustment during the training process, and the test set is used for final evaluation of the generalization ability of the model.

[0138] Step S2111: The model is iteratively trained using the training set, and the model performance is verified by the validation set after each iteration. When the validation indicators of the validation set are all stable for a fixed number of consecutive times, the training is stopped. The validation indicators include pest and disease type identification accuracy, organ lesion positioning accuracy, and lesion matching accuracy at different growth stages.

[0139] Step S21111: The training set is subjected to data enhancement processing, including random horizontal flipping, random vertical flipping, random rotation within a fixed angle range, random cropping of a fixed proportion of the area, and random adjustment of the brightness within a fixed range, so that the number of samples in the expanded training set is a fixed multiple of the original number.

[0140] The training set is subjected to data enhancement processing to increase the diversity of samples and improve the generalization ability of the model. Data enhancement operations include random horizontal flipping, with a 50% probability of flipping the image horizontally; random vertical flipping, also with a certain probability; random rotation within a fixed angle range, such as random rotation between -30 degrees and 30 degrees; random cropping of a fixed proportion of the area, randomly cropping a certain proportion of the area from the image as a new sample; and random adjustment of the brightness within a fixed range. Through the above data enhancement operations, the number of samples in the expanded training set is a fixed multiple of the original number, such as 3 times, which enriches the training data.

[0141] Step S21112: The expanded training set is divided into training batches of a fixed size, each containing a fixed number of samples, and the samples are evenly distributed according to pest and disease type, organ type, and growth stage.

[0142] The expanded training set is divided into multiple training batches of a fixed size, each containing a fixed number of samples, such as 32 samples. When dividing the batches, ensure that the samples in each batch are evenly distributed according to pest and disease type, organ type, and growth stage, to avoid too much uniformity in the types of samples in a batch. This allows the model to learn diverse features in each iteration, improving the training effect.

[0143] Step S21113: Initialize the weight parameters of each layer of the model, and initialize the weights of the convolutional and fully connected layers. The initial weight distribution is adjusted according to the statistical law of plant lesion features, and the bias parameters are initialized to a fixed small value.

[0144] Initialize the weight parameters of each layer of the model. For the weights of the convolutional and fully connected layers, initialize them according to the statistical law of plant lesion features. For example, by analyzing the feature distribution of a large number of plant lesion images, adjust the initial distribution of the weights to make the initial weights closer to the direction of the optimal solution. Initialize the bias parameters of all layers to a fixed small value, such as 0.01, to avoid zero output of neurons and help stabilize the network startup.

[0145] Step S21114: Set the training parameters, set the total number of iterations to a fixed value, set the initial value of the learning rate to a fixed value, use an adaptive learning rate adjustment strategy, when the validation index of the validation set does not improve for a fixed number of times in succession, reduce the learning rate by a fixed proportion, and set the minimum learning rate to a fixed proportion of the initial value.

[0146] The parameters of model training are set. The total number of iterations is set to a fixed large value, such as 100 times. The initial value of the learning rate is set to a fixed value, such as 0.001. An adaptive learning rate adjustment strategy is used, such as the adaptive learning rate strategy of the Adam optimizer. When the validation index (such as accuracy) of the validation set does not improve for a fixed number of times (such as 5 times) in succession, the learning rate is reduced by a fixed proportion (such as 0.1 times) to fine-tune the parameters. The minimum learning rate is set to a fixed proportion (such as 0.0001 times) of the initial value to prevent the learning rate from being too low and causing the training to stagnate.

[0147] Step S21115: Start the iterative training, input each training batch in turn each time, input the batch samples into the model, and obtain the feature outputs of each branch through the parallel processing of the lesion texture detail extraction branch, the lesion morphological feature extraction branch, and the lesion color feature extraction branch; After processing by the lesion feature interaction layer and the lesion feature aggregation layer, the lesion aggregation feature is obtained, and the probability value corresponding to each pest disease type is output through the pest disease type matching layer.

[0148] The iterative training process of the model is started. In each iteration, each training batch is input into the model in turn. The batch samples are first processed by the feature input layer and converted into branch input features. Then, the branch input features are input into the three feature extraction branches in parallel, and the lesion texture feature map, the lesion morphological feature map, and the lesion color feature vector are obtained through convolution, pooling, batch normalization, etc. of each branch. The above feature outputs are input into the lesion feature interaction layer, and the interaction feature map is obtained through spatial and channel attention weighting processing. The interaction feature map is input into the lesion feature aggregation layer, and the fixed-length lesion aggregation feature vector is obtained through deepening extraction of the convolutional neural network block and processing of the adaptive pooling layer. Finally, the lesion aggregation feature vector is input into the pest disease type matching layer, and the probability value corresponding to each pest disease type is output through the mapping of the two fully connected layers and the processing of the probability activation function.

[0149] Step S21116: Calculate the loss value between the output probability value and the sample labeled pest disease type using a cross-entropy loss function, which is calculated based on the correspondence between the probability value of each sample and the true pest disease type. The position label corresponding to the true pest disease type is set to 1, and the labels of other types are set to 0. The loss value is propagated back to each layer of the model using a backpropagation algorithm, and the weight parameters and bias parameters of each layer are adjusted using a gradient descent optimization algorithm. The step size of gradient descent is determined by the current learning rate.

[0150] For each training batch, the loss value between the output probability value of the model and the true pest disease type labeled by the sample is calculated using a cross-entropy loss function. The calculation of the cross-entropy loss function is based on the difference between the predicted probability distribution of each sample and the true label distribution. The position label corresponding to the true pest disease type is set to 1, and the labels of other types are set to 0, forming a one-hot vector. The larger the loss value, the greater the difference between the prediction and the true situation. The loss value is propagated back to each layer of the model using a backpropagation algorithm, and the gradient of the loss value with respect to the weight parameters and bias parameters of each layer is calculated. Using a gradient descent optimization algorithm, the weight parameters and bias parameters of each layer are adjusted based on the calculated gradient and the current learning rate, so that the loss value gradually decreases. The step size of gradient descent is determined by the current learning rate, and the larger the learning rate, the larger the step size and the faster the parameter update.

[0151] Step S21117: After each iteration period, input the validation set into the trained model to obtain the pest disease type recognition result, lesion positioning result, and growth stage matching result of the validation set. Calculate the pest disease type recognition accuracy, which is the proportion of correctly recognized samples to the total number of samples in the validation set. Calculate the organ lesion positioning accuracy, which is the proportion of samples whose lesion positioning coordinates and labeled coordinates are within the allowed range to the total number of samples in the validation set. Calculate the lesion matching accuracy for different growth stages, which is the proportion of correctly matched numbers of the same lesion association group in different growth stages to the total number of lesion association groups in the validation set.

[0152] After each iteration period (i.e., all training batches are trained once), input the validation set into the model in the current training state to obtain the pest disease type recognition result, lesion positioning result (coordinates of lesions on organs), and growth stage matching result of the validation set. Calculate the validation indicators. The pest disease type recognition accuracy is the proportion of correctly recognized samples to the total number of samples in the validation set. The organ lesion positioning accuracy is the proportion of samples whose lesion positioning coordinates and labeled coordinates are within the allowed range (e.g., pixel distance less than a certain value) to the total number of samples in the validation set. The lesion matching accuracy for different growth stages is the proportion of correctly matched numbers of the same lesion association group in different growth stages to the total number of lesion association groups in the validation set. The above indicators evaluate the performance of the model from different angles.

[0153] Step S21118: record the validation indicators of each iteration period, and determine whether the fluctuation ranges of the validation indicators of a continuous fixed number of times are all lower than the respective fluctuation thresholds. If all the fluctuation ranges are lower than the fluctuation thresholds, it is determined that the validation indicators remain stable, and the training is stopped.

[0154] The validation indicator values of each iteration period, such as the pest and disease type recognition accuracy, organ lesion positioning accuracy, and lesion matching accuracy at different growth stages, are recorded. The fluctuation threshold of each validation indicator is set. When the fluctuation ranges (i.e., the absolute values of the differences between adjacent two iterations) of all the validation indicators in a continuous fixed number of iterations (such as 3 times) are all lower than the respective fluctuation thresholds, it is determined that the validation indicators remain stable, and the model converges. At this time, the model training is stopped, and the current model parameters are saved as the final training result.

[0155] Step S2112: the trained model is tested for performance using the test set. If the pest and disease type recognition accuracy, organ lesion positioning accuracy, and lesion matching accuracy at different growth stages of the test set all meet the set standard, the construction of the plant pest and disease specific deep learning model is completed. If not, the network parameters of each branch are adjusted and retrained until the test is passed.

[0156] The trained model is tested for performance using an independent test set. The test set samples are input into the model to obtain the test results, and the pest and disease type recognition accuracy, organ lesion positioning accuracy, and lesion matching accuracy at different growth stages of the test set are calculated. The above test indicators are compared with the set standard. If the accuracy rate is above 90%, for example, the construction of the plant pest and disease specific deep learning model is completed. If the standard is not met, the test results are analyzed to find out the reasons for the insufficient model performance, such as low accuracy rate of some pest and disease type recognition, inaccurate positioning of specific organs, etc. According to the analysis results, the network parameters of each branch are adjusted, such as increasing the number of convolution layers, adjusting the hole rate, changing the number of neurons in the fully connected layer, etc. Then the training set and the validation set are used to retrain and validate the model until all the indicators of the test set meet the set standard.

[0157] Based on the same inventive concept, please refer to Figure 2 The structure schematic block diagram of the deep learning based plant pest and disease image recognition system 100 provided by the embodiments of the present application for executing the above deep learning based plant pest and disease image recognition method is shown in FIG. 1. The deep learning based plant pest and disease image recognition system 100 can include a communication unit 110, a machine readable storage medium 120, and a processor 130.

[0158] In this embodiment, the machine readable storage medium 120 and the processor 130 are both located in the plant pest and disease image recognition system 100 based on deep learning and are separately arranged. Alternatively, the machine readable storage medium 120 can also be integrated into the processor 130, and can communicate and interact with an external system through the communication unit 110. The machine readable storage medium 120 is used to store machine executable instructions for implementing the scheme of the present application, and the processor 130 is used to execute the machine executable instructions stored in the machine readable storage medium 120 to implement the plant pest and disease image recognition method based on deep learning provided by the foregoing method embodiments.

[0159] It should be noted that, in order to simplify the description of the present application and to facilitate the understanding of one or more embodiments of the present application, in the foregoing description of the embodiments of the present application, various features are sometimes incorporated into one embodiment, figure or description thereof.

Claims

1. A deep learning-based image recognition method for plant insect diseases, characterized in that, The method includes: Acquire a multi-source image set of the plant at multiple growth stages. The multi-source image set includes an overall image of the plant, a partial image of the leaf, a partial image of the stem, and time markers and shooting angle markers for each growth stage. Based on the growth stage time markers and shooting angle markers, a plant image temporal association network is constructed. The multi-source image set is then subjected to organ-level temporal alignment processing to obtain an organ-level temporal aligned image set and organ association coefficients between images. By using the organ correlation coefficient, cross-growth stage lesion feature association processing is performed on the organ-level time-aligned image set to obtain a cross-stage lesion association set and the temporal evolution information of lesions on different organs. The cross-stage lesion association set is input into a preset plant insect disease-specific deep learning model for feature aggregation and type matching to obtain lesion aggregation features and preliminary insect disease type results. By combining the temporal evolution information of the lesions on different organs, the preliminary results of the insect and disease type are verified by organ adaptation, and the final plant insect and disease identification results and temporal evolution map of the lesion organs are generated.

2. The plant insect and disease image recognition method based on deep learning according to claim 1, characterized in that, The process involves constructing a plant image temporal correlation network based on the growth stage time markers and shooting angle markers, performing organ-level temporal alignment processing on the multi-source image set, and obtaining an organ-level temporally aligned image set and inter-image organ correlation coefficients, including: Extract the growth stage time identifier and shooting angle identifier information of each image in the multi-source image set, perform primary sorting of all images according to the order of growth stages, and perform secondary sorting according to the shooting angle identifier information within the same growth stage to obtain the initial time-series angle image sequence. Plant organ contour features are extracted from each image in the initial time-series angle image sequence. The plant organ contour features include main stem contour features, leaf contour features and stem branch contour features. The main stem contour features include the coordinates of the main stem contour points and the contour curvature change value. The leaf contour features include the coordinates of the leaf contour points, the direction of the leaf vein contour and the density of the leaf edge serrations. The stem branch contour features include the coordinates of the branch contour points and the connection angle between the branch and the main stem. Using each image as a network node, and the similarity of plant organ contour features between images taken from the same shooting angle at adjacent growth stages as initial edge weights, an initial temporal association network is constructed. The initial edge weights are calculated by weighting the standardized values ​​of the main stem contour overlap ratio, the leaf contour overlap ratio, and the branch connection angle deviation value. The branch connection angle deviation value is converted into a deviation score that is comparable to the contour overlap ratio. The higher the main stem contour overlap ratio, the higher the leaf contour overlap ratio, and the higher the branch connection angle deviation score, the larger the initial edge weight. Calculate the indirect association weights of images with the same shooting angle in non-adjacent growth stages in the initial temporal association network. The indirect association weights are determined by the sum of the initial edge weights of all paths between two nodes. The path selection rule is that the interval between the growth stages corresponding to the nodes in the path does not exceed N, and the shooting angle identification information is consistent, where N is a set positive integer. The initial edge weights and indirect association weights are integrated into organ association coefficients between nodes to obtain the temporal association network of plant images. The larger the organ association coefficient, the stronger the association between the corresponding organs in the two images. Based on the organ correlation coefficient in the plant image temporal correlation network, a reference image with an organ correlation coefficient higher than the correlation threshold is selected. The main stem axis and leaf vein line in the reference image are used as calibration benchmarks to perform coordinate calibration on the corresponding organ in the current image, so that the overlap of the main benchmark lines of the same organ in different images reaches the calibration standard, and an organ-level temporal aligned image set is obtained.

3. The plant insect and disease image recognition method based on deep learning according to claim 1, characterized in that, The process of associating lesion features across growth stages using the organ correlation coefficient on the organ-level temporally aligned image set yields a cross-stage lesion association set and temporal evolution information of lesions in different organs, including: Plant organ segmentation processing is performed on each image in the organ-level time-aligned image set. A segmentation method based on plant organ morphology templates is used to segment the image into leaf region images, main stem region images, and branch region images. The coordinate range information and organ type identifier of each organ region image are recorded. Preliminary lesion detection processing is performed on the images of each organ region. A sliding window detection method adapted to the morphology of plant organs is adopted. The window shape is determined according to the organ type. An elliptical window is used for the leaf region, a rectangular window is used for the main stem region, and a slender window is used for the branch region. The window is moved to traverse the organ region image according to a set step size, and the gray-scale change features and color distribution features within the window are extracted. Image samples of healthy organs of the same plant variety are collected, and gray-scale variation features and color distribution features of corresponding organ regions in the image samples are extracted. The mean range and distribution pattern of the sample features are calculated to determine the normal feature range of each organ. Window regions whose grayscale variation features or color distribution features deviate from the normal feature range of the corresponding organ are selected as suspected lesion regions. The location coordinates, size information and corresponding feature data of each suspected lesion region are recorded to form an organ suspected lesion information table for each image. The feature data includes color distribution histogram data, texture distribution feature data and shape parameter data. Based on the organ correlation coefficient, related images with organ correlation coefficients higher than the correlation threshold of the current image are selected, and the suspected lesion information table of the organ type identifier in the related images is extracted as the related lesion information. The location correspondence and feature data similarity between the suspected lesion areas in the current image and the suspected lesion areas in the associated images are compared. The location correspondence is determined by the overlap ratio of the coordinates of the two suspected lesion areas in the organ coordinate system. The feature data similarity is calculated by comprehensively considering the overlap of the color distribution histogram, the matching ratio of the texture distribution features, and the standardized value of the deviation of the shape parameters. The deviation of the shape parameters is converted into a deviation score that is comparable to the overlap and matching ratio. The higher the overlap of the color distribution histogram, the higher the matching ratio of the texture distribution features, and the higher the deviation score of the shape parameters, the higher the feature data similarity. Suspected lesion areas that meet the association criteria in terms of both location correspondence and feature data similarity are divided into the same lesion association group. Each lesion association group contains associated suspected lesion areas on the same organ at different growth stages. Extract the location coordinate change trajectory, size change trend and feature data change pattern of each suspected lesion area in each lesion association group, and generate temporal evolution information of lesions in different organs. The temporal evolution information includes the location offset trajectory of the lesion in the organ coordinate system, the size expansion or contraction curve and the feature data change matrix. All lesion association groups and the suspected lesion regions contained in each group are combined into a cross-stage lesion association set. The association records the temporal evolution information of the lesions in different organs and the corresponding organ type identifiers of each group.

4. The plant insect and disease image recognition method based on deep learning according to claim 1, characterized in that, The process of constructing and optimizing the pre-defined deep learning model specific to plant pests and diseases includes: A basic model architecture is constructed, which includes a feature input layer, a parallel plant lesion feature extraction branch, a lesion feature interaction layer, a lesion feature aggregation layer, an insect and disease type matching layer, and a dynamic optimization module. The parallel plant lesion feature extraction branches are a lesion texture detail extraction branch, a lesion morphology feature extraction branch, and a lesion color feature extraction branch. Configure the network parameters for the lesion texture detail extraction branch, which includes multiple convolutional layers, multiple pooling layers, and two batch normalization layers. The kernel size of the multiple convolutional layers increases sequentially, and the stride remains fixed. The pooling layers use max pooling, and the pooling kernel size is set to a fixed value. The batch normalization layers are inserted between two adjacent convolutional layers. Configure the network parameters of the lesion morphology feature extraction branch. The lesion morphology feature extraction branch includes multiple dilated convolutional layers, one global average pooling layer and one batch normalization layer. The dilation rate of the dilated convolutional layers increases sequentially at a fixed ratio. The pooling kernel size of the global average pooling layer is set to a fixed value. The batch normalization layer is inserted after the middle dilated convolutional layer. Configure the network parameters of the lesion color feature extraction branch, which includes multiple convolutional layers, multiple fully connected layers and one random dropout layer. The number of neurons in the fully connected layers decreases sequentially, and the dropout probability of the random dropout layer is set to a fixed value and inserted between two adjacent fully connected layers. The feature interaction strategy of the lesion feature interaction layer is configured as follows: the feature map output by the parallel plant lesion feature extraction branch is spliced ​​according to the channel dimension, a spatial weight map is generated through the spatial attention mechanism, the spliced ​​feature map is spatially weighted through the spatial weight map, a channel weight vector is generated through the channel attention mechanism, and the spatially weighted feature map is channel-weighted through the channel weight vector to obtain the interaction feature map. Configure the aggregation method of the lesion feature aggregation layer. The aggregation method is to input the interactive feature map output by the feature interaction layer into the convolutional neural network block. The convolutional neural network block contains multiple convolution operations to deepen the extraction of comprehensive lesion features, fuse lesion features of different dimensions, and optimize feature expression. Then, the feature map is converted into a fixed-length lesion aggregation feature vector through an adaptive pooling layer. Configure the output strategy of the pest and disease type matching layer. The pest and disease type matching layer contains two fully connected layers and a probabilistic activation function. The first fully connected layer maps the fixed-length lesion aggregate feature vector to the feature vector of the intermediate dimension. The second fully connected layer maps it to the dimension consistent with the number of known plant pest and disease types. The probabilistic activation function converts the output into the probability value corresponding to each pest and disease type. Configure the optimization rules of the dynamic optimization module. The optimization rules are to collect the deviation data between the probability value of the insect and disease type output by the model and the insect and disease type labeled in the sample in real time. When the deviation value of a fixed number of consecutive samples is higher than the deviation threshold, the model parameters are fine-tuned. Collect labeled sample sets containing different types of plant pests and diseases. The labeled sample sets contain lesion images of each type of pest and disease in different plant organs and at different growth stages. Each sample is labeled with the name of the pest and disease type, the type of organ where the lesion is located, the growth stage of the lesion, and the severity level of the lesion. The labeled sample set is divided into training set, validation set and test set according to a fixed ratio. The training set contains samples of different insect and disease types, different organs and different growth stages. The sample distribution of the validation set and test set is consistent with that of the training set. The model is trained iteratively using the training set. After each iteration, the model performance is verified using the validation set. Training is stopped when the validation metrics of the validation set remain stable for a fixed number of consecutive iterations. The validation metrics include the accuracy of identifying insect and disease types, the accuracy of locating organ lesions, and the accuracy of matching lesions at different growth stages. The trained model is tested using a test set. If the accuracy of disease and pest identification, organ lesion localization, and lesion matching at different growth stages all meet the set standards, the construction of the plant disease and pest-specific deep learning model is complete. If not, the network parameters of each branch are adjusted and the model is retrained until the test is passed.

5. The plant insect and disease image recognition method based on deep learning according to claim 1, characterized in that, The step of inputting the cross-stage lesion association set into a preset plant pest and disease-specific deep learning model for feature aggregation and type matching processing to obtain lesion aggregation features and preliminary pest and disease type results includes: Each lesion association group in the cross-stage lesion association set is preprocessed, and image data of each suspected lesion area in each group is extracted. The image size is adjusted according to the organ type where the lesion is located. Leaf lesion images are adjusted to a uniform rectangular size, and main stem and branch lesion images are adjusted to a uniform elongated size. The adjusted image data is normalized by pixel value normalization, converting the pixel values ​​into a fixed range to obtain a standardized lesion image sequence. The temporal arrangement information of the standardized lesion image sequence is extracted, and the growth stage interval data in the temporal evolution information of the lesion on different organs is combined to perform temporal completion processing on the standardized lesion image sequence. When the image interval between adjacent growth stages exceeds the interval threshold, an interpolation method based on the evolution law of organ lesions is used to generate an intermediate image to obtain a complete temporal lesion image sequence. The complete temporal sequence of lesion images is input into the feature input layer of a deep learning model specifically for plant pests and diseases. Through the dimension transformation operation of the input layer, the image sequence is converted into an input format suitable for each branch of the model to obtain the branch input features. The branch input features are used to extract the lesion texture details. Through the sequential processing of the preset convolutional layer, pooling layer and batch normalization layer, the surface texture details of the lesion in the image are extracted to obtain the lesion texture feature map. The branch input features are simultaneously input into the lesion morphology feature extraction branch. The overall morphological features of the lesion are extracted through the expansion of the receptive field of the dilated convolutional layer and the global information aggregation of the global average pooling layer, resulting in a lesion morphology feature map. The input features of the branch are synchronously input into the lesion color feature extraction branch. Through color feature extraction of the convolutional layer and feature transformation processing of the fully connected layer, the color distribution features and color gradient features of the lesion are extracted to obtain the lesion color feature vector. The lesion texture feature map, lesion morphology feature map, and lesion color feature vector are input into the lesion feature interaction layer. First, the lesion texture feature map and lesion morphology feature map are concatenated by channel. A spatial weight map is generated through a spatial attention mechanism for spatial weighting. Then, the weighted feature map and lesion color feature vector are concatenated by channel. A channel weight vector is generated through a channel attention mechanism for channel weighting to obtain the interaction feature map. The interactive feature map is input into the lesion feature aggregation layer. Through the in-depth extraction of plant lesion-specific convolutional neural network blocks and the dimension unification processing of the adaptive pooling layer, a fixed-length lesion aggregation feature is obtained. The lesion aggregation features are input into the insect and disease type matching layer. Through feature mapping of two fully connected layers and probability transformation of the probability activation function, the probability value corresponding to each insect and disease type is obtained. The insect and disease type with the highest probability value is selected as the preliminary insect and disease type result of the lesion association group.

6. The plant insect and disease image recognition method based on deep learning according to claim 1, characterized in that, The preliminary insect-disease type results are then subjected to organ adaptation verification processing by combining the temporal evolution information of the lesions on different organs, generating the final plant insect-disease identification results and a temporal evolution map of the lesion organs, including: Extract the temporal evolution information of lesions in different organs for each lesion association group in the cross-stage lesion association set, and analyze the position offset trajectory, size expansion or contraction curve and feature data change matrix of the suspected lesion area in the organ coordinate system at each growth stage; Obtain a preset temporal evolution feature library of plant insect diseases and organs. The temporal evolution feature library of plant insect diseases and organs contains standard temporal evolution rules of various known plant insect diseases and pests on different organs. Each insect disease and pest type corresponds to a set of standard evolution rules, including standard position offset trajectory, standard size change curve and standard feature data change rules on leaves. The standard evolution rules on the main stem and branches are similar. Based on the organ type identifier corresponding to the lesion association group, the standard temporal evolution law of each insect and disease type under the organ type is matched from the plant insect and disease organ temporal evolution feature library. The positional offset trajectory of the lesion association group is compared with the matched standard positional offset trajectory, and the trajectory matching degree is calculated. Compare the size expansion or contraction curve with the standard size change curve to calculate the size matching degree; compare the feature data change matrix with the standard feature data change pattern to calculate the feature matching degree. Organ adaptation weights were assigned to trajectory matching degree, size matching degree, and feature matching degree. The comprehensive temporal matching degree was obtained by weighted summation. The trajectory matching degree of leaf lesions had the highest weight, the size matching degree of main stem lesions had the highest weight, and the feature matching degree of branch lesions had the highest weight. If the overall temporal matching degree is higher than the organ adaptation verification threshold, then the preliminary insect and disease type result of the lesion association group is confirmed as a valid insect and disease type result. If the overall temporal matching degree is not higher than the organ adaptation verification threshold, the lesion aggregation features of the lesion association group, the corresponding organ type identifier, and the growth stage information are input into the dynamic optimization module. After the model parameters are fine-tuned, the type matching is performed again to obtain the secondary identification type result. The organ adaptation verification steps are repeated until the effective pest and disease type result is obtained. Collect all effective insect and disease type results of lesion association groups, associate them with the corresponding growth stage time markers, organ type markers, organ region coordinate range information and feature data of suspected lesion regions, and construct a prediction association table, which contains a unique identifier for each record. A map coordinate system is constructed with growth stage as the horizontal axis and plant organ type as the vertical axis. The horizontal axis is labeled in the order of growth stage, and the vertical axis is labeled in the order of leaf, main stem, and branch. In the map coordinate system, the outline of each organ is drawn according to the coordinate range information of the organ region. The shape of the outline is consistent with the actual shape of the corresponding organ. Within the outline of each organ, lesion markers are drawn according to the location coordinates of the suspected lesion area. The color of the lesion marker corresponds one-to-one with the effective pest and disease type results. The size of the lesion marker is proportional to the size of the suspected lesion area. The lesion marker contains a unique identifier for the associated group of the lesion. In the atlas, lesion markers at different growth stages of the same lesion association group are connected by broken lines. The thickness of the broken lines is proportional to the organ association coefficient of the lesion association group. Key nodes of feature data changes are marked next to the broken lines. Add a legend to one side of the map, indicating the color corresponding to different effective pest and disease types, the size range corresponding to the size of the lesion identifier, the range of organ correlation coefficients corresponding to the thickness of the broken line, and the meaning of the unique identifier. Combine the prediction correlation table with the completed map to form the final plant pest and disease identification result.

7. The plant insect and disease image recognition method based on deep learning according to claim 2, characterized in that, The extraction of plant organ contour features from each image in the initial temporal angle image sequence includes: Each image in the initial time-series angle image sequence is processed by color space conversion to extract the hue channel image. The hue channel image is then segmented by thresholding to divide it into the plant foreground region and the background region, resulting in a binarized image. After performing morphological closing operations on the binarized image, morphological opening operations are performed to obtain an optimized binarized image. Edge detection is then performed on the optimized binarized image to form an initial set of contour segments. The initial set of contour segments is subjected to contour screening. Based on the morphological characteristics of plant organs, screening rules are set to select the main stem contour segments, leaf contour segments, and stem branch contour segments. The selected main stem contour segments are then fitted. A cubic polynomial fitting algorithm is used to fit the segments into a smooth curve that conforms to the upright shape of the plant main stem. The coordinates of the sampling points on the smooth curve are extracted at fixed intervals as the coordinates of the main stem contour points. The curvature change value at each sampling point is calculated as the curvature change value of the main stem contour, thus forming the main stem contour feature. The selected leaf contour segments are closed, and the unclosed segments are extended to intersect to form a closed contour. The coordinates of the sampling points on the closed contour are extracted as the coordinates of the leaf contour points. The contour direction of the main vein and lateral veins of the leaf is extracted by the leaf vein detection algorithm. The number and distribution density of the serrated line segments on the leaf edge are counted as the leaf edge serration density, which constitutes the leaf contour features. The selected stem branch contour lines are fitted to obtain the coordinates of the branch contour points. The angle between the branch contour line and the main stem contour curve at the connection point is calculated by the angle calculation algorithm as the connection angle between the branch and the main stem, thus forming the stem branch contour features. The outline features of the main stem, leaves, and stem branches are combined into the outline features of the plant organs in each image, and the coordinates of the connection points between branches and the main stem, as well as the coordinates of the connection points between leaves and branches, are recorded.

8. The plant insect and disease image recognition method based on deep learning according to claim 3, characterized in that, The preliminary lesion detection processing for each organ region image employs a sliding window detection method adapted to plant organ morphology, extracting grayscale variation features and color distribution features within the window, including: The parameters of the sliding window are determined based on the organ type identifier. An elliptical window is used for the leaf region, with the major axis length set to a fixed proportion of the longest side of the leaf region image, and the minor axis length set to a fixed proportion of the major axis length. A rectangular window is used for the main stem region, with the window length and width set to a fixed proportion of the main stem region image length and width. A long, narrow window is used for the branching region, with the window length and width set to a fixed proportion of the branching region image length and width. The sliding step size is uniformly set to a fixed proportion of the shortest side of the corresponding window, determined through testing with multiple normal images of the same organ type. Move the window with a set sliding step size, traverse the entire organ region image in order from left to right and from top to bottom, and record the position coordinates of each window and the image pixel data within the window; Gray-level change features are extracted from the pixel data within each window. The pixel data is converted into a grayscale image, and the standard deviation of gray values, the mean of gray value gradient, and the gray value entropy of the grayscale image within the window are calculated. The standard deviation of gray values, the mean of gray value gradient, and the gray value entropy of the grayscale image are combined in a fixed order to form gray-level change features. The standard deviation of gray values ​​reflects the dispersion of gray-level distribution within the window, the mean of gray value gradient reflects the drasticness of gray-level changes within the window, and the gray value entropy reflects the uniformity of gray-level distribution within the window. Color distribution features are extracted from the pixel data in each window, and the pixel data is converted into a hue, saturation and lightness color space. Histograms of the hue channel, saturation channel and lightness channel are calculated respectively. Each histogram is divided into a fixed number of intervals. The proportion of pixels in each interval to the total number of pixels in the window is calculated. The interval pixel proportions of the three channel histograms are combined in channel order to form the color distribution feature. Multiple healthy images of the same plant species and organ type were collected as normal samples. Each normal sample image was processed with the same window parameters to extract the grayscale change features and color distribution features of each window. Calculate the mean and fluctuation range of each window feature of the same organ type in normal samples, and take the fluctuation range of the mean plus or minus a fixed multiple as the normal feature range of the organ type. The gray scale change feature and the color distribution feature correspond to their respective normal feature ranges.

9. The plant insect and disease image recognition method based on deep learning according to claim 4, characterized in that, The process of iteratively training the model using a training set, validating the model's performance using a validation set after each iteration, and stopping training when all validation metrics on the validation set remain stable for a fixed number of consecutive iterations includes: The training set is subjected to data augmentation processing, including random horizontal flipping, random vertical flipping, random rotation within a fixed angle range, random cropping of a fixed proportion area, and random adjustment of brightness within a fixed range, so that the number of training set samples after augmentation is a fixed multiple of the original number. The expanded training set is divided into training batches according to a fixed batch size. Each training batch contains a fixed number of samples, which are evenly distributed according to insect and disease type, organ type, and growth stage. Initialize the weight parameters of each layer of the model, and initialize the weights of the convolutional layer and the fully connected layer. In particular, adjust the initial weight distribution according to the statistical law of plant lesion characteristics, and initialize the bias parameter to a fixed small value. Set the training parameters, set the total number of iterations to a fixed value, set the initial value of the learning rate to a fixed value, and adopt an adaptive learning rate adjustment strategy. When the validation metric of the validation set does not improve for a fixed number of consecutive times, the learning rate is reduced by a fixed percentage, and the minimum learning rate is set to a fixed percentage of the initial value. Start iterative training. In each iteration, each training batch is input sequentially. The batch samples are input into the model. Through parallel processing of the lesion texture detail extraction branch, the lesion morphology feature extraction branch, and the lesion color feature extraction branch, the feature output of each branch is obtained. After processing by the lesion feature interaction layer and the lesion feature aggregation layer, the lesion aggregation features are obtained, and then the probability values ​​corresponding to each insect and disease type are output through the insect and disease type matching layer. The cross-entropy loss function is used to calculate the loss value between the output probability value and the insect disease type labeled in the sample. The calculation of the cross-entropy loss function is based on the correspondence between the probability value of each sample and the actual insect disease type. The position label corresponding to the actual insect disease type is set to 1, and other types are set to 0. The loss value is backpropagated to each layer of the model using the backpropagation algorithm. The weight and bias parameters of each layer are adjusted using the gradient descent optimization algorithm. The step size of the gradient descent is determined by the current learning rate. After each iteration cycle, the validation set is input into the trained model to obtain the insect and disease type identification results, lesion location results, and growth stage matching results of the validation set. Calculate the accuracy rate of insect and disease type identification, which is the proportion of correctly identified samples to the total number of samples in the validation set; calculate the accuracy rate of organ lesion localization, which is the proportion of samples whose lesion localization coordinates deviate from the labeled coordinates within the allowable range to the total number of samples in the validation set. Calculate the lesion matching accuracy at different growth stages, that is, the proportion of the number of correctly matched lesion association groups at different growth stages to the total number of lesion association groups in the validation set. Record the verification metrics for each iteration cycle, and determine whether the fluctuation range of the verification metrics for a fixed number of consecutive cycles is lower than their respective fluctuation thresholds. If they are all lower than the fluctuation thresholds, the verification metrics are determined to be stable, and training is stopped. If they are not, the next iteration continues until the conditions for stopping training are met.

10. A plant insect and disease image recognition system based on deep learning, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the deep learning-based plant pest and disease image recognition method according to any one of claims 1 to 9 by executing the machine-executable instructions.

Images