Corn kernel mildew identification method based on image recognition

Abstract

The application discloses a corn kernel mildew recognition method based on image recognition and belongs to the technical field of agricultural product detection and image processing. The method comprises the following steps: acquiring an original color image and a near-infrared image of corn kernels to be recognized; performing color space conversion on the color image to extract a chroma saturation component image, and performing radiation correction on the near-infrared image to generate a moisture distribution image; pixel-level fusion is performed to construct a mildew-sensitive feature tensor; a pre-trained deep convolution network is used to encode the feature tensor to obtain a plurality of hierarchical feature maps; the structural similarity index of adjacent hierarchical feature maps is calculated to generate a significant mildew area heat map; an adaptive threshold segmentation is performed to obtain an initial mildew area mask; the initial mask and the original color image are superimposed, the mildew area boundary is optimized by means of a conditional random field, and finally, a mildew recognition result is output. The method deeply fuses multi-modal information and can realize accurate recognition of the mildew area of corn kernels.

Description

Technical Field

[0001] This invention relates to the field of agricultural product testing and image processing technology, specifically to a method for identifying mold growth in corn kernels based on image recognition. Background Technology

[0002] Corn is highly susceptible to mold growth during storage and distribution, causing significant economic losses and posing a serious threat to human and animal health due to the toxins produced. Rapid and accurate identification of moldy areas in corn kernels is crucial for ensuring food quality and safety. Existing methods for identifying mold in corn kernels have long relied on manual visual inspection and chemical testing. Manual inspection is subjective and inefficient, while chemical testing is time-consuming and destructive. Non-destructive testing methods based on machine vision are gradually being applied, with visible light color image analysis distinguishing moldy kernels from normal kernels by extracting features such as color and texture. However, corn kernels vary widely, and the natural pigmentation of the endosperm and seed coat, along with color changes caused by early mold growth, are difficult to reliably separate in the red-green-blue color space. Relying solely on visible light information can easily lead to misjudgments. Furthermore, near-infrared imaging technology can reflect the internal moisture state of the kernel, and mold activity is usually accompanied by localized increases in moisture. However, using only moisture distribution maps for threshold discrimination is insufficient to distinguish between physiological moisture increases caused by mold growth and changes in environmental humidity or differences in the kernel's own moisture content, resulting in high rates of missed detections and false alarms. Fusing visible light images and near-infrared moisture information is an improved approach. Most existing fusion methods simply stitch or weight the images at the feature or decision layers, failing to fully explore pixel-level complementary information. They also underutilize the joint features of color saturation reduction and moisture enrichment caused by mold growth, and cannot construct a collaborative feature space that is highly sensitive to mold growth.

[0003] When segmenting fused images using deep learning, conventional semantic segmentation networks directly encode and decode the input image. The lack of explicit modeling of the structural consistency of moldy regions across different scale feature layers results in segmentation results with blurry edges, making it difficult to accurately fit the moldy boundaries. Post-processing optimization using conditional random fields requires a reliable initial saliency map as guidance, but existing methods struggle to generate saliency heatmaps with good regional consistency based on the structural relationships between multi-level feature maps, limiting the final segmentation accuracy. There is an urgent need to address the technical challenges of effectively integrating color saturation and moisture anomaly information at the pixel level, and of locating structurally clear salient moldy regions from multi-scale deep features and finely optimizing their boundaries. Summary of the Invention

[0004] This paper presents a method for identifying mold growth in corn kernels based on image recognition. It can deeply integrate the color saturation features of color images with the moisture distribution features of near-infrared images and generate mold growth region identification results with fine boundaries, thereby overcoming the problems of insufficient expression of mold growth sensitive features and rough boundary segmentation of mold growth regions in existing methods.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for identifying moldy corn kernels based on image recognition. By fusing the color saturation information of the color image of corn kernels with the moisture distribution information of the near-infrared image, and combining deep learning and probabilistic graphical models, the method achieves high-precision identification and boundary optimization of moldy areas in corn kernels.

[0006] This method first acquires the original color image and near-infrared image of the corn kernel sample to be identified. The original color image undergoes color space conversion to extract the chroma and saturation components. As a preferred embodiment, the original color image is converted from a red-green-blue color space to a hue-saturation-lightness color space. The hue and saturation components are separated from the converted space, and median filtering is applied to both components for noise reduction. The denoised hue and saturation components are then multiplied pixel by pixel to obtain a chroma and saturation component image that highlights the color variation characteristics caused by mold. The median filter window size is preferably set to 5×5 pixels to maintain the clarity of the mold boundary while removing noise. Radiometric correction is then performed on the near-infrared image to generate a moisture distribution map. Preferably, the radiance value of each pixel in the near-infrared image is collected. Based on the pre-calibrated linear regression model between moisture content and radiance value, the radiance value is converted into an initial moisture estimate. The initial moisture estimate is then normalized using the global mean as a benchmark to obtain a normalized moisture value. Finally, a moisture distribution map is generated by arranging the pixels according to their spatial positions. This distribution map can accurately reflect the spatial changes in moisture content inside the grain caused by mold growth.

[0007] A mold-sensitive feature tensor is constructed by pixel-level fusion of the chroma / saturation component image and the moisture distribution image. Specifically, the chroma / saturation component image and the moisture distribution image are adjusted to the same spatial resolution, preferably using a bicubic interpolation algorithm to ensure image smoothness. At each pixel location, the pixel values ​​of the two images are concatenated into a two-dimensional feature vector. Then, all the feature vectors at all pixel locations are stacked in the row and column order of the original image to form a mold-sensitive feature tensor with two channels. This feature tensor integrates two complementary mold characterization information—color distortion and moisture anomaly—enhancing the sensitivity to early or latent mold growth.

[0008] A pre-trained deep convolutional network is used to encode the mold-sensitive feature tensor, resulting in multi-level feature maps. As a preferred implementation, this feature tensor is input into a ResNet-50 model pre-trained on the ImageNet dataset, passing it sequentially through multiple convolutional blocks. Each convolutional block outputs an intermediate feature map, whose spatial size decreases and the number of channels increases with increasing block depth. The intermediate feature maps output from the last three convolutional blocks are extracted and used as shallow, intermediate, and top-level feature maps, respectively, to represent semantic information at different scales. Shallow features preserve rich spatial details, while deep features contain advanced mold-related semantics.

[0009] Based on the extracted multi-level feature maps, the structural similarity index between adjacent level feature maps is calculated, and a heat map of significant moldy areas is generated accordingly. Preferably, the deepest level, the highest-level feature map, is bilinearly upsampled to match the intermediate level feature maps. Figure 1 To obtain a first similarity map, the structural similarity index of each local window of the two layers is calculated based on their spatial dimensions. Then, the intermediate layer feature map is bilinearly upsampled to match the shallow layer feature map. Figure 1 By calculating the similarity of spatial dimensions, a second similarity map is obtained. Finally, the first and second similarity maps are averaged pixel-wise to generate a heatmap. This heatmap can accurately locate abnormal tissue structures caused by mold growth by capturing cross-scale feature consistency changes and effectively suppress background noise.

[0010] An adaptive threshold segmentation is performed on the heatmap to obtain an initial moldy region mask. Preferably, an intensity histogram is generated by statistically analyzing the intensity values ​​of all pixels in the heatmap, and the optimal segmentation threshold for this histogram is calculated using Otsu's method. Pixels with intensity values ​​greater than the threshold are marked as suspected moldy pixels, while pixels with intensity values ​​less than or equal to the threshold are marked as background pixels. Further connected component analysis is performed on the marking results. Specifically, all connected components are extracted using the eight-neighbor connectivity rule, and the total number of pixels in each connected component is calculated as its area. All pixels in isolated connected components with areas smaller than a preset minimum moldy area are forcibly marked as background pixels, while connected components with areas greater than or equal to this preset area are retained, forming the initial moldy region mask. This process automatically determines the optimal segmentation criteria and eliminates minor false detection regions introduced by noise, improving the robustness of the segmentation results.

[0011] The initial moldy region mask is overlaid on the original color image, and the boundary of the moldy region is optimized using a conditional random field (CRF) to output the final moldy identification result. As a preferred approach, the initial mask is used as the initial label for the CRF model. Each pixel in the original color image is used as an observation node, and edge connections are constructed between adjacent pixels. When defining the univariate potential function, the label value of each pixel is read from the initial mask, and the percentage of pixels labeled as moldy within a local window centered on that pixel is counted as the moldy confidence level. The negative logarithm of the confidence level is taken to obtain the univariate potential function value when labeled as moldy, and the negative logarithm of the value minus the confidence level is taken to obtain the univariate potential function value when labeled as non-moldy. When defining the binary potential function, it is calculated based on the color difference and spatial distance between adjacent pixels in the original color image, making pixels with similar colors and spatial proximity more likely to be assigned the same label. By performing mean-field inference on the CRF model, the label of each pixel is iteratively updated until convergence, resulting in the optimized moldy region mask as the final identification result. This step utilizes the low-level color and spatial information of the image to finely adjust the initial segmentation boundary, ensuring that the boundary of the moldy area in the final output closely matches the actual moldy outline of the corn kernels, significantly improving the accuracy and completeness of the recognition results.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: The original color image and near-infrared image of the corn kernel sample to be identified were acquired. The original color image was converted from a red-green-blue color space to a hue-saturation-lightness color space, separating the hue and saturation components. After median filtering for noise reduction, these components were multiplied pixel by pixel to obtain a hue-saturation component map. This component map highlights the purity of the kernel surface color; moldy areas typically exhibit a significant decrease in saturation due to mycelial erosion and pigment degradation. Simultaneously, radiometric correction was performed on the near-infrared image. Based on a pre-calibrated linear regression model of moisture content and radiance values, the radiance value of each pixel was converted into an initial moisture estimate, which was then normalized using the global mean to generate a moisture distribution map. This map accurately depicts the localized high moisture state caused by mold. At each pixel location, the pixel values ​​of the hue-saturation component map and the moisture distribution map were concatenated to construct a two-channel mold-sensitive feature tensor. This tensor simultaneously encodes two complementary physical quantities sensitive to mold growth—color saturation and moisture content—at the pixel level. Moldy regions exhibit a coupling of low saturation and high moisture content, contrasting sharply with the high saturation or low moisture characteristics of normal tissue. By inputting this joint feature into a deep network, replacing single-modality image information, the network can directly learn the significant discriminative boundary between moldy and normal tissue in the feature space. This eliminates artifacts and interference introduced by variations in corn variety, grain morphology, and ambient light, significantly improving the accuracy and robustness of mold detection.

[0013] A pre-trained residual network model is used to perform multi-level feature encoding on the mold-sensitive feature tensor, sequentially outputting hierarchical feature maps generated by multiple deep convolutional blocks. The intermediate feature maps of the last three convolutional blocks, with decreasing spatial size and increasing channel number at each layer, are extracted. The deepest high-level feature map is bilinearly upsampled to the same size as the intermediate layer feature map, and the structural similarity index between the two is calculated at each local window to obtain the first similarity map. Then, the intermediate layer feature map is upsampled to the same size as the shallow layer feature map. Figure 1 The second similarity map is calculated; finally, the two similarity maps are averaged pixel-wise to generate a heatmap of significant moldy areas. The structural similarity index integrates measures of local brightness, contrast, and structural correlation, ensuring that regions with stable moldy structural features shared by different levels of abstraction in the feature maps receive high responses, while chaotic structures appearing only at a single scale or caused by noise are suppressed. The resulting heatmap can fully highlight the overall morphology of the moldy area and effectively smooth the internal response, eliminating false alarms caused by non-target textures in the background. Then, the Otsu method is used to adaptively determine the segmentation threshold and remove small-area isolated connected components to obtain an initial moldy area mask. The initial mask is used as the initial label for the conditional random field model, and each pixel of the original color image is introduced as an observation node. A univariate potential function based on local moldy confidence and a binary potential function based on the color difference and spatial distance between adjacent pixels are defined. The label of each pixel is iteratively updated through mean field inference. Color difference drives the boundary curve to converge to a position where the image gradient is significant, while spatial distance constraints ensure regional connectivity. The final output moldy region mask has a smooth and delicate boundary, eliminating the jagged edges and internal holes commonly found in the initial mask, and obtaining high-precision moldy recognition results. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart of a method for identifying moldy corn kernels based on image recognition; Figure 2 This is a flowchart of the chroma and saturation component image extraction process; Figure 3 This is a flowchart of near-infrared image radiometric correction and moisture distribution map generation; Figure 4 This is a flowchart of the construction process for the mold-sensitive feature tensor; Figure 5 It is a frequency distribution map of pixel values ​​for hue component, saturation component, and chroma-saturation component. Figure 6 It is a linear regression calibration curve of moisture content and near-infrared radiance; Figure 7 It is a two-dimensional scatter plot of the feature vectors sensitive to mold growth; Figure 8 It consists of a multi-layer feature map structure similarity index and a heat map curve of a significantly moldy area; Figure 9 This is a schematic diagram of the heat map intensity histogram and the optimal segmentation threshold of the Otsu method. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] See Figure 1 This invention provides a method for identifying mold growth in corn kernels based on image recognition. The method includes the following steps: acquiring the original color image and near-infrared image of the corn kernel sample to be identified; performing color space conversion on the original color image to extract the chroma saturation component map, and performing radiometric correction on the near-infrared image to generate a moisture distribution map; fusing the chroma saturation component map and the moisture distribution map at the pixel level to construct a mold-sensitive feature tensor; using a pre-trained deep convolutional network to perform feature encoding on the mold-sensitive feature tensor to obtain feature maps at multiple levels; calculating the structural similarity index between adjacent feature maps, and generating a heat map of significant moldy regions based on the structural similarity index; performing adaptive threshold segmentation on the heat map to obtain an initial moldy region mask; superimposing the initial moldy region mask on the original color image, optimizing the boundary of the moldy region using a conditional random field, and outputting the final mold identification result.

[0018] Example 1: In specific implementation, please refer to Figure 2 The following is the implementation method for performing color space conversion on the original color image and extracting the chroma and saturation component map.

[0019] After acquiring the original color image of the corn kernel sample to be identified, the original color image is converted from the red-green-blue color space to the hue-saturation-lightness color space. In the red-green-blue color space, each pixel is represented by red, green, and blue channel values. The hue-saturation-lightness color space consists of hue, saturation, and lightness components. During the conversion, the red, green, and blue channel values ​​are first mapped from their original integer range to a normalized interval. Then, based on the numerical relationships between the normalized values ​​of the three channels, the hue, saturation, and lightness values ​​are calculated pixel by pixel. After the color space conversion is completed, an image in the hue-saturation-lightness color space is obtained.

[0020] The hue and saturation components are separated from the converted hue-saturation-lightness color space. The hue component is a single-channel image where the grayscale value of each pixel corresponds to the hue value of that pixel in the hue-saturation-lightness color space. The saturation component is another single-channel image where the grayscale value of each pixel corresponds to the saturation value of that pixel.

[0021] Median filtering is applied to both the hue and saturation components for noise reduction. The median filtering window size is 5×5 pixels. For the hue component, a 5×5 pixel neighborhood window is defined centered on each pixel in the hue component image. All twenty-five hue values ​​within the window are read, sorted by value, and the median value is selected as the filtered hue value for that pixel. For pixels in the edge regions of the hue component image, the image boundaries are mirrored to ensure each pixel has a complete 5×5 pixel neighborhood window. The same method is used for the saturation component: for each pixel in the saturation component image, twenty-five saturation values ​​within a 5×5 pixel neighborhood window are taken, sorted, and the median value is selected as the filtered saturation value. After these processes, the denoised hue and saturation components are obtained.

[0022] The denoised hue component and the denoised saturation component are multiplied pixel by pixel to obtain the hue / saturation component map. The multiplication process uses the following formula: in, This indicates the position of the chroma saturation component in the graph. Line 1 The pixel value of the column pixel. This indicates the hue component located at the _____th ... Line 1 The hue value of the column pixels, This indicates the saturation component located at the th position after noise reduction. Line 1 The saturation values ​​of each pixel are calculated. For each pixel location, the hue value of that location in the denoised hue component is multiplied by the saturation value of the same location in the denoised saturation component. The product is used as the pixel value at the corresponding location in the chroma-saturation component map. After multiplying all pixel locations, the chroma-saturation component map is constructed.

[0023] See Figure 5 In the graph, the horizontal axis represents pixel values, ranging from 0 to 1, and the vertical axis represents the frequency of the corresponding pixel value. The solid curve represents the frequency distribution of pixel values ​​for the hue component, the dashed curve represents the frequency distribution of pixel values ​​for the saturation component, and the dotted-dash curve represents the frequency distribution of pixel values ​​for the chroma and saturation component graph (i.e., the image after multiplying the hue and saturation components).

[0024] The solid curve of the frequency distribution of the hue component shows that pixel values ​​are mainly concentrated in the range of 0.1 to 0.4, with a peak at approximately 0.3 and a frequency exceeding 40. This indicates that most pixels in the hue component image have low hue values ​​and exhibit a certain degree of concentration. The dashed curve of the saturation component shows that pixel values ​​are relatively evenly distributed, concentrated in the higher range of 0.4 to 0.9, with a peak at approximately 0.6 to 0.7. The frequency fluctuates significantly, indicating that the saturation component image as a whole presents high saturation values ​​and rich variations.

[0025] The dotted-line curves of the chroma and saturation component map differ significantly from those of the individual hue and saturation components. The peak pixel frequency is mainly concentrated between 0 and 0.15, with a peak frequency exceeding 70, exhibiting a clear left-skewed distribution. This aligns with the pixel-by-pixel multiplication of the hue component after median filtering and noise reduction with the saturation component, as described in Example 1. The multiplication operation results in more smaller pixel values ​​when lower hue and saturation values ​​are multiplied, thus enhancing the frequency of the low pixel value range.

[0026] Overall, this figure reflects the pixel value distribution characteristics of the hue and saturation components after color space transformation and median filtering noise reduction, as well as the distribution pattern of pixel values ​​in the resulting chroma-saturation component image. The pixel values ​​in the chroma-saturation component image are concentrated in the lower value range, highlighting areas with relatively weak color and saturation in the corn kernel image. This provides an effective color feature foundation for subsequent pixel-level fusion with the moisture distribution map and the construction of a mold-sensitive feature tensor. This data is consistent with the description of color space transformation and median filtering in Example 1, demonstrating the important role of the chroma-saturation component image in the feature extraction stage before mold identification.

[0027] Example 2: In specific implementation, please refer to Figure 3 The process of performing radiometric correction on near-infrared images to generate a moisture distribution map is as follows.

[0028] First, the radiance value of each pixel in the near-infrared image is acquired. The near-infrared image is obtained by a near-infrared imaging device; each pixel location corresponds to a radiance value, which is recorded as a digital signal. The entire near-infrared image is traversed, and the digital signal value at each pixel location is read one by one as the corresponding pixel's radiance value.

[0029] Based on a pre-calibrated linear regression model of moisture content and radiance values, the radiance value of each pixel is converted into an initial moisture estimate. The pre-calibrated linear regression model is established as follows: multiple graded moisture content corn kernel calibration samples are selected, and the true moisture content of each calibration sample is obtained using standard moisture determination methods; using the same near-infrared imaging equipment and conditions as those used to acquire the near-infrared images of the samples to be identified, near-infrared images of each calibration sample are acquired, and the average radiance value of the central region of each calibration sample image is extracted as the calibration radiance value; the true moisture content of all calibration samples is used as the dependent variable, and the corresponding calibration radiance value is used as the independent variable, and least-squares linear fitting is performed to obtain the linear relationship between moisture content and radiance values, where the linear regression model is expressed as: in, Indicates the position located in the image. Line 1 Initial moisture estimate of column pixels, in weight percentage; Indicates the position located in the image. Line 1 The radiance value of each column pixel is represented by a digital signal value; The linear regression slope coefficient obtained through calibration has a value range that depends on the response characteristics of the near-infrared imaging device and is obtained by fitting the aforementioned calibration sample data; The linear regression intercept coefficients obtained through calibration are also obtained by fitting the calibration sample data. For each pixel in the near-infrared image of the corn kernel sample to be identified, the radiance value of the corresponding pixel is substituted into the above linear regression model to calculate the initial moisture estimate of each pixel.

[0030] Using the global mean of the initial moisture estimates as a benchmark, the initial moisture estimate for each pixel is normalized to obtain a normalized moisture value. When calculating the global mean, the initial moisture estimates of all pixels in the entire initial moisture estimate image are summed and divided by the total number of pixels to obtain the global mean of the initial moisture estimates, denoted as . Then, each pixel is normalized by dividing the initial moisture estimate of each pixel by the global mean to obtain the normalized moisture value. After normalization, the overall mean of the normalized water content is 1, which can eliminate the overall intensity differences between different samples caused by lighting and imaging parameter fluctuations.

[0031] The normalized moisture values ​​are arranged according to their pixel spatial location to generate a moisture distribution map. The moisture distribution map is a single-channel image with the same size as the near-infrared image. Line 1 The grayscale value of a column pixel is the normalized water content of the corresponding pixel location. .

[0032] See Figure 6 In the figure, the horizontal axis represents the radiance value of the corresponding pixel in the near-infrared image of corn kernels, ranging from approximately 2000 to 4000, in digital signal values. The vertical axis represents the moisture content of the corresponding pixel, in weight percentage, ranging from approximately 8% to 42%. The calibration sample data points distributed as hollow circles represent multiple known moisture contents and corresponding radiance values. The distribution of these points along the diagonal of the figure clearly reflects the correlation between radiance and moisture content. The solid line is the linear regression model curve between moisture content and radiance obtained by fitting the calibration sample data using the least squares method, showing the linear increase in moisture content with radiance. This linear regression model effectively covers the radiance range from 2000 to 4000, corresponding to the moisture content range from 10% to 40%. The fitted curve closely matches the calibration sample points, indicating that the model accurately reflects the linear relationship between near-infrared radiance and corn kernel moisture content. This figure illustrates the technical solution in Example 2, which establishes a linear regression model between radiance and moisture content through pre-calibration, providing a theoretical basis and data support for converting the radiance of near-infrared image pixels into initial moisture estimates.

[0033] Example 3: In specific implementation, please refer to Figure 4 The process of constructing a mold-sensitive feature tensor by pixel-level fusion of the chroma saturation component map and the moisture distribution map is as follows.

[0034] A bicubic interpolation algorithm is used to adjust the chroma / saturation component image and the moisture distribution image to the same spatial resolution. Before adjustment, the image height and width of the chroma / saturation component image and the moisture distribution image are obtained simultaneously. The target spatial resolution of both images is uniformly set to the larger of their image height and width, ensuring that the adjusted images have the same image height and width. For the image requiring resizing, at each desired pixel location in the target image, the bicubic interpolation algorithm calculates the pixel value based on the pixel values ​​of the 16 closest known pixels in the source image. Taking a desired pixel location as an example, 16 known pixels within a 4×4 neighborhood of the desired pixel location are identified in the source image. The pixel value of the desired pixel is calculated using a weighted formula based on the interpolation kernel function. During the interpolation process, the bicubic interpolation kernel function performs convolution processing on the horizontal and vertical distances between the desired pixel location and the known pixel locations. The interpolation kernel function reaches a maximum value of 1 when the distance is zero, decreases when the distance is 1 and 2, and becomes zero when the distance is greater than 2. After bicubic interpolation, the chromaticity saturation component map and the water distribution map have the same image height and image width, that is, they have the same spatial resolution.

[0035] At each pixel location, the pixel values ​​of the chroma / saturation component map and the pixel values ​​of the water distribution map are concatenated to form a two-dimensional feature vector. This vector is then used to define the first feature vector in the resolution-adjusted image. Line 1 Column pixel position, read the pixel value of the chroma / saturation component map at the corresponding position, and record it as . Simultaneously, the pixel values ​​at the corresponding locations on the moisture distribution map are read and recorded as follows: .Will As the first element of the two-dimensional feature vector, As the second element of the two-dimensional feature vector, it constitutes the two-dimensional feature vector. ,in satisfy: in, Indicates the first Line 1 Two-dimensional feature vectors constructed at column pixel positions. This represents the first chromaticity component in the bicubic interpolation graph. Line 1 The pixel value of the column pixel. This represents the first element in the water distribution map after bicubic interpolation. Line 1 The column represents the pixel values, with square brackets indicating that two elements are arranged sequentially to form a vector. This concatenation operation is performed on every pixel in the image.

[0036] The two-dimensional feature vectors of all pixel locations are stacked in row and column order of the original image to form a two-channel feature tensor, which serves as the mold-sensitive feature tensor. During stacking, the height and width of the mold-sensitive feature tensor are set to the height of the uniformly sized image, and the number of channels is 2. The first channel of the mold-sensitive feature tensor stores the first element of the two-dimensional feature vector at each pixel location. In the second channel of the mold-sensitive feature tensor, the second element of the two-dimensional feature vector at each pixel location is stored. The stacked mold-sensitive feature tensor fully preserves the spatial correspondence between the chroma saturation component map and the moisture distribution map, with the two channels at each pixel location fusing color features and moisture features respectively.

[0037] See Figure 7 The figure shows the scatter distribution of the mold-sensitive feature vector, where the horizontal axis represents the chroma saturation component value, ranging from approximately 0 to 0.9, and the vertical axis represents the moisture distribution value, ranging from approximately 0.55 to 1.5. Each point in the scatter plot is a two-dimensional feature vector sample constructed in Example 3, that is, a feature vector formed by combining the pixel values ​​of the chroma saturation component map and the moisture distribution map at the corresponding pixel positions after bicubic interpolation processing.

[0038] Observation of the scatter plot distribution shows that the color saturation component values ​​are mainly concentrated between 0.1 and 0.7, exhibiting a relatively uniform distribution, while the moisture distribution values ​​are concentrated between 0.8 and 1.2, showing some fluctuation. The overall scatter plot shows no obvious clustering trend, indicating that the color saturation and moisture characteristics are spatially complementary and can comprehensively reflect the color and moisture information of corn kernels.

[0039] The data features in this figure reflect the distribution of the two-dimensional feature vector formed by concatenating the two single-channel feature maps through pixel values ​​in the feature space during the pixel-level fusion process described in Example 3. This provides a foundation for subsequent deep convolutional networks to extract multi-level features. Through this scatter distribution, the differences in color saturation and moisture characteristics between moldy and non-moldy areas can be further understood, thereby supporting the process of feature encoding and salient region generation using a pre-trained network in Example 4.

[0040] Example 4: In the specific implementation, a pre-trained deep convolutional network is used to encode the mold-sensitive feature tensor to obtain feature maps at multiple levels. The structural similarity index between adjacent feature maps is calculated, and a heat map of significant moldy areas is generated based on the structural similarity index. The implementation method is as follows.

[0041] The mold sensitivity feature tensor was input into a ResNet-50 model pre-trained on the ImageNet dataset. The ResNet-50 model is a residual network architecture containing an initial convolutional layer and sixteen convolutional blocks. Each convolutional block consists of multiple convolutional layers, batch normalization layers, and modified linear unit activation layers. Features are transferred between convolutional blocks via residual connections. The weight parameters of the pre-trained ResNet-50 model were obtained through training on a classification task on the ImageNet dataset using stochastic gradient descent optimization. The initial learning rate was set to 0.1, and it was decayed to one-tenth of its original value every 30 training epochs. The weight decay coefficient was set to 0.0001, and the momentum coefficient was set to 0.9. The total number of training epochs was 90. After pre-training, the weights of each layer of the ResNet-50 model were saved and used for subsequent feature encoding.

[0042] After the mold sensitivity feature tensor is input into the ResNet-50 model, it passes through multiple convolutional blocks of the ResNet-50 model sequentially, with each convolutional block outputting an intermediate feature map. The height of the input mold sensitivity feature tensor is denoted as... Width is denoted as The number of channels is 2. After the initial convolutional layer and the first convolutional block, the spatial size of the output intermediate feature map is reduced to half the input size, and the number of channels becomes 64. Subsequently, for each subsequent convolutional block, the spatial size of the intermediate feature map is reduced by half based on the spatial size of the intermediate feature map output by the previous convolutional block, and the number of channels is doubled based on the number of channels of the intermediate feature map output by the previous convolutional block.

[0043] The intermediate feature maps from the outputs of the last three convolutional blocks in the ResNet-50 model are extracted to serve as feature maps for multiple layers. The deepest layer is designated as the top-layer feature map, the intermediate layers as intermediate-layer feature maps, and the shallowest layer as shallow-layer feature maps. The top-layer feature map corresponds to the output of the last convolutional block in the ResNet-50 model, the intermediate-layer feature maps correspond to the output of the second-to-last convolutional block, and the shallow-layer feature maps correspond to the output of the third-to-last convolutional block. The top-layer feature map has the smallest spatial size. The number of channels is 2048; the spatial size of the intermediate layer feature map is... The number of channels is 1024; the spatial size of the shallow feature map is... The number of channels is 512.

[0044] The highest-layer feature map is bilinearly upsampled to make its spatial size similar to that of the intermediate-layer features. Figure 1To. During bilinear upsampling, for each pixel position to be determined in the upsampled highest-level feature map, the corresponding floating-point coordinates are found in the original highest-level feature map. The feature values ​​of the four nearest integer coordinates surrounding the floating-point coordinates are taken. First, two linear interpolations are performed in the horizontal direction, and then one linear interpolation is performed in the vertical direction to obtain the feature value of that position in the upsampled highest-level feature map. The spatial size of the upsampled highest-level feature map becomes... The spatial dimensions are exactly the same as those of the intermediate layer feature maps.

[0045] The structural similarity index is calculated for each local window of the upsampled highest-layer feature map and intermediate-layer feature maps to obtain the first similarity map. During calculation, an 11×11 pixel sliding window is used, traversing the entire feature map with a step size of 1 pixel. For each local window position, all pixel values ​​of all channels within that window in the upsampled highest-layer feature map are extracted to form the first local vector, and all pixel values ​​of all channels within the same window position in the intermediate-layer feature map are extracted to form the second local vector. The similarity between the two local vectors is calculated using the structural similarity index calculation formula: in, Indicates the position in the first similarity map. The structural similarity index value at the location, and These represent the row and column indices of the local window center on the feature map, respectively. This represents the mean value of all pixel values ​​in the current local window of the highest-level feature map after upsampling. This represents the mean value of all pixel values ​​in the intermediate layer feature map within the current local window. This represents the variance of all pixel values ​​in the current local window of the highest-level feature map after upsampling. This represents the variance of all pixel values ​​in the intermediate layer feature map within the current local window. This represents the covariance of pixel values ​​between the highest-level feature map and the intermediate-level feature map within the current local window after upsampling. The first stability constant has a value of [value missing]. ,in The dynamic range of the eigenvalues ​​is taken here. ; The second stability constant has a value of The structural similarity index value calculated for each local window position is placed at the corresponding center position of the first similarity map. After all local windows are calculated, the complete first similarity map is obtained.

[0046] Bilinear upsampling is performed on the intermediate layer feature maps to make their spatial dimensions similar to those of the shallow layer features. Figure 1 The implementation of bilinear upsampling is the same as the aforementioned upsampling methods, but the spatial size of the intermediate layer feature map is reduced from... Become .

[0047] The structural similarity index between the upsampled intermediate layer feature map and the shallow layer feature map is calculated to obtain the second similarity map. The calculation method is the same as that used to obtain the first similarity map, employing an 11×11 pixel sliding window and traversing the feature map with a step size of 1 pixel. At each local window position, all pixel values ​​within the window of the upsampled intermediate layer feature map and the shallow layer feature map are extracted. The similarity degree is calculated using the structural similarity index calculation formula, with the first stability constant used in the calculation. Second stability constant The values ​​are consistent with those mentioned above. After calculation, a second similarity mapping map is obtained.

[0048] The heatmap of significantly moldy areas is obtained by averaging the first and second similarity maps pixel by pixel. For each pixel location... Read the structural similarity index value of that location in the first similarity map. Read the structural similarity index value of that location in the second similarity map. The arithmetic mean of the two values ​​is calculated as a heat map of the significantly moldy area at location. The intensity value at a given location. The heatmap of a significantly moldy area is a single-channel image. Areas with higher intensity values ​​indicate greater structural differences in the feature maps of adjacent layers at that location, and thus a stronger likelihood of mold growth.

[0049] See Figure 8 In the graph, the horizontal axis represents the pixel location index, ranging from 0 to 500, and the vertical axis represents the structural similarity index, with a value ranging from approximately 0.4 to 0.95. The curves include the first similarity map represented by the solid line, the second similarity map represented by the dashed line, and the heat map of significant moldy areas represented by the dotted line; all three fluctuate with changes in pixel location.

[0050] The structural similarity index of the first similarity map is generally at a high level, mostly between 0.7 and 0.95, indicating strong local structural features. However, at approximately pixels 70, 190, 310, and 450, the first similarity map shows a significant drop to valleys of approximately 0.55 to 0.6, reflecting increased structural differences between multi-level feature maps in the corresponding regions.

[0051] The curve of the second similarity map is generally stable, with the index value mostly distributed between 0.6 and 0.9. It also shows a clear downward trend at the above pixel positions, with the valley value slightly below 0.5, indicating that the structural similarity between the intermediate layer and the shallow layer feature map is reduced at these positions.

[0052] The overall trend of the heatmap curves for significantly moldy areas lies between the first and second similarity maps, with overall values ​​ranging from 0.6 to 0.9. Significant downward peaks and troughs are also observed at pixel positions approximately 70, 190, 310, and 450, corresponding to the troughs in the structural similarity index. This indicates significant differences in the multi-layer feature map structures at these locations, suggesting potential mold-sensitive areas.

[0053] Overall, the three curves exhibit synchronized downward fluctuations at multiple key pixel locations, reflecting significantly different structural segments in the feature maps extracted by the deep convolutional network at different levels. This trend effectively maps the spatial distribution characteristics of the moldy region, providing a basis for subsequent generation of mold heatmaps and localization of salient regions. This figure visually demonstrates the calculation process of the structural similarity index between feature maps at different levels in Example 4 and its application effect in moldy region identification.

[0054] Example 5: In the specific implementation, the heat map is subjected to adaptive threshold segmentation to obtain an initial moldy area mask. The initial moldy area mask is then superimposed on the original color image. The boundary of the moldy area is optimized using a conditional random field, and the final moldy recognition result is output. The implementation method is as follows.

[0055] The intensity values ​​of all pixels in the heatmap are statistically analyzed to generate an intensity histogram. The heatmap is a single-channel image, and the intensity value of each pixel represents the response intensity of a significantly moldy area at that location. The process involves iterating through all pixels in the heatmap, recording the intensity value of each pixel, dividing the intensity value range into 256 equal intervals, and summing the number of pixels falling into each interval to generate the intensity histogram.

[0056] The Otsu method is used to calculate the optimal segmentation threshold of the intensity histogram. The Otsu method iterates through all candidate intensity values ​​in the intensity histogram. For each candidate intensity value, the heatmap pixels are divided into foreground and background classes. The inter-class variance between the foreground and background classes is calculated, and the candidate intensity value that maximizes the inter-class variance is selected as the optimal segmentation threshold.

[0057] Pixels with intensity values ​​greater than the optimal segmentation threshold in the heatmap are marked as suspected mold pixels, while pixels with intensity values ​​less than or equal to the optimal segmentation threshold are marked as background pixels. For each pixel location in the heatmap, the intensity value of the corresponding pixel is compared with the optimal segmentation threshold, and a temporary label of mold or background is assigned to the pixel based on the comparison result.

[0058] Connectivity analysis was performed on the marked suspected moldy pixels. The marked suspected moldy pixels were scanned, and all connected components were extracted using the eight-neighbor connectivity rule. The eight-neighbor connectivity rule means that for any suspected moldy pixel, if other suspected moldy pixels exist in its eight neighboring positions (up, down, left, right, and four diagonals), they are grouped into the same connected component. Starting from the top left corner of the image, the scan proceeded line by line. When an unvisited suspected moldy pixel was encountered, a connected component growth process was initiated, using breadth-first search or depth-first search to mark all suspected moldy pixels connected to that pixel through eight-neighbor paths as belonging to the same connected component. This process was repeated until all suspected moldy pixels were assigned to a single connected component. The total number of pixels in each connected component was calculated as the area of ​​that connected component. All pixels in connected components with an area smaller than a preset minimum moldy area were forcibly marked as background pixels. The preset minimum moldy area was set to 100 pixels, a value calculated based on the minimum physical size of the moldy corn kernel area in the image and the imaging resolution, effectively eliminating isolated false positives caused by imaging noise or local illumination changes. Retain connected regions with an area greater than or equal to the preset minimum moldy area. Pixels within these connected regions continue to retain suspected moldy markings. The combination of all pixel markings forms the initial moldy region mask.

[0059] The initial moldy region mask is used as the initial label for the Conditional Random Field (CRF) model. Each pixel in the original color image is used as an observation node in the CRF model, with the number of observation nodes equal to the total number of pixels in the original color image. Edges are constructed between adjacent pixels, defined as pairs of four neighboring pixels sharing an edge on the image plane; that is, each pixel has an edge with the pixels above, below, to the left, and to the right of it. After construction, an undirected graph structure for the CRF model is formed, where each node corresponds to a pixel, and each edge connects a pair of adjacent pixels.

[0060] Define a univariate potential function. Read the label value of each pixel from the initial moldy region mask, categorizing them as moldy or non-mold. For each pixel, calculate the proportion of pixels with a moldy label within a local window centered on that pixel, relative to the total number of pixels in the window. Set the local window to 5×5 pixels; this size effectively covers the local moldy feature area of ​​a single corn kernel in the image while avoiding excessive background interference. This proportion is used as the mold confidence score for the corresponding pixel. The mold confidence score ranges from 0 to 1; a higher score indicates stronger initial evidence that the pixel is moldy. The univariate potential function is calculated using the following formula: in, Indicates the index of a pixel; Indicates the value of the label; Label indicating mold growth; Indicates non-mildew label; Represents pixels The confidence level of mold growth, i.e., pixels The label value within the 5x5 local window is the ratio of the number of moldy pixels to the total number of pixels in the window (25). When pixels... When a pixel is labeled as moldy, the univariate potential function value is the negative natural logarithm of the mold confidence level; when the pixel... When assigned a non-mold label, the univariate potential function value is the negative natural logarithm of the result after subtracting the mold confidence level.

[0061] A binary potential function is defined, calculated based on the color difference and spatial distance between adjacent pixels in the original color image. The original color image is converted from the red-green-blue color space to the CIELAB color space, which contains one luminance channel and two color-opposite channels. The conversion process first involves linearization mapping followed by nonlinear transformation. For a pair of adjacent pixels connected by an edge, the luminance component, green-red component, and blue-yellow component values ​​of this pair of pixels in the CIELAB color space are read, and the color difference is calculated as the Euclidean distance between the three pixel values. Simultaneously, the positions of this pair of adjacent pixels in the image's row and column coordinates are read, and the spatial distance is calculated as the Euclidean distance between the two pixel coordinates. The binary potential function is a combination of kernel functions, consisting of two weighted Gaussian kernels. The first Gaussian kernel takes spatial distance as a single variable, while the second Gaussian kernel takes the joint variable of spatial distance and color difference as input. When both the spatial distance and color difference between adjacent pixels are small, the binary potential function tends to encourage this pair of pixels to be assigned the same label; when the color difference is large, the binary potential function allows this pair of pixels to be assigned different labels. The specific expression of the binary potential function includes a weighting coefficient and a kernel bandwidth parameter. The weighting coefficient is used to balance the contributions of the two Gaussian kernels, and the kernel bandwidth parameter controls the influence range of spatial distance and color difference.

[0062] The Conditional Random Field (CRF) model is subjected to mean-field inference, and the label of each pixel is iteratively updated until convergence. During initialization, a univariate potential function is used to set the approximate edge probability of each pixel belonging to the moldy label and the non-moldy label. In each iteration, a message passing step is first performed: for each pixel, the edge probabilities of all its adjacent pixels from the previous round are collected, and the message passed from the adjacent pixels is calculated using the univariate potential function; then a weighted update step is performed: the univariate potential function is summed with all messages from adjacent pixels, and the updated edge probability is obtained through softmax normalization. This iterative process is repeated until the maximum change in the probability of all pixel labels between two adjacent iterations is less than a preset convergence threshold, or the maximum number of iterations (10) is reached. The maximum number of iterations is set to 10 based on a combination of the image resolution and the convergence characteristics of the CRF model, ensuring boundary optimization while controlling computational time. After convergence, the label with the larger final edge probability of each pixel is taken as the final label of that pixel. The final labels of all pixels are combined to form the optimized moldy region mask, and the optimized moldy region mask is taken as the final moldy recognition result.

[0063] See Figure 9 The horizontal axis of the graph represents the intensity value of the heatmap, ranging from 0 to 1, while the vertical axis represents the pixel frequency corresponding to that intensity value. The graph uses black bars to illustrate the frequency distribution of the heatmap intensity values, exhibiting a clear bimodal structure. The first peak on the left is located in the intensity value range of approximately 0.1 to 0.3, with a pixel frequency as high as nearly 90, indicating that most pixels have low intensity and belong to the background area. The second peak on the right is located in the intensity value range of approximately 0.6 to 0.95, with a highest frequency of approximately 37, reflecting that some pixels have a higher response intensity, possibly corresponding to moldy areas. The optimal segmentation threshold, marked by a black dashed line at approximately an intensity value of 0.42, is the optimal threshold automatically calculated from the intensity histogram using the Otsu method. This threshold effectively distinguishes between low-intensity background pixels on the left and high-intensity suspected moldy pixels on the right, achieving adaptive binary segmentation of the heatmap. Using this threshold, pixels with intensity values ​​higher than 0.42 are marked as suspected moldy pixels, while pixels with values ​​lower than or equal to this value are marked as background, thus forming an initial moldy area mask. This figure accurately reflects the process of adaptive threshold segmentation of the heat map and the selection of key parameters in Example 5, providing basic data support for subsequent conditional random field boundary optimization.

[0064] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for identifying mold growth in corn kernels based on image recognition, characterized in that, The method includes: Acquire the original color and near-infrared images of the corn kernel sample to be identified; The original color image is subjected to color space conversion, the chroma and saturation component map is extracted, and the near-infrared image is radiometrically corrected to generate a moisture distribution map. The color saturation component map and the moisture distribution map are fused at the pixel level to construct a mold-sensitive feature tensor. The mold-sensitive feature tensor is encoded using a pre-trained deep convolutional network to obtain feature maps at multiple levels. Calculate the structural similarity index between adjacent hierarchical feature maps, and generate a heat map of significant moldy areas based on the structural similarity index; Adaptive threshold segmentation is performed on the heat map to obtain an initial moldy region mask; The initial moldy area mask is superimposed on the original color image, and the boundary of the moldy area is optimized using a conditional random field to output the final moldy recognition result. The specific implementation method for generating a moisture distribution map by performing radiometric correction on the near-infrared image is as follows: Collect the radiance value of each pixel in the near-infrared image; Based on a pre-calibrated linear regression model of moisture content and radiance value, the radiance value of each pixel is converted into an initial moisture estimate. Based on the global mean of the initial moisture estimate, the initial moisture estimate of each pixel is normalized to obtain the normalized moisture value. The normalized moisture values ​​are arranged according to pixel spatial positions to generate the moisture distribution map; The linear regression model is expressed as follows: ； in, Indicates the position located in the image. Line 1 Initial moisture estimate of column pixels, in weight percentage; Indicates the position located in the image. Line 1 The radiance value of each column pixel is represented by a digital signal value; The linear regression slope coefficient obtained through calibration has a range of values ​​that depends on the response characteristics of the near-infrared imaging device and is obtained by fitting the aforementioned calibration sample data. The linear regression intercept coefficients are obtained through calibration and are fitted using the calibrated sample data.

2. The method for identifying moldy corn kernels based on image recognition according to claim 1, characterized in that, The specific implementation method for performing color space conversion on the original color image and extracting the chroma and saturation component map is as follows: The original color image is converted from the red-green-blue color space to the hue-saturation-brightness color space; The hue component and saturation component are separated from the converted hue-saturation-brightness color space; The hue component and the saturation component are respectively subjected to median filtering for noise reduction. Then, the denoised hue component and the saturation component are multiplied by pixel to obtain the hue and saturation component map.

3. The method for identifying moldy corn kernels based on image recognition according to claim 2, characterized in that, The median filter has a filter window size of 5×5 pixels.

4. The method for identifying moldy corn kernels based on image recognition according to claim 1, characterized in that, The specific implementation method for constructing the mold-sensitive feature tensor by pixel-level fusion of the chromaticity saturation component map and the moisture distribution map is as follows: Adjust the chromaticity saturation component map and the moisture distribution map to the same spatial resolution; At each pixel location, the pixel value of the chromaticity saturation component map and the pixel value of the moisture distribution map are concatenated to form a two-dimensional feature vector. The two-dimensional feature vectors of all pixel locations are stacked in the row and column order of the original image to form a feature tensor with two channels, which serves as the mold-sensitive feature tensor.

5. The method for identifying moldy corn kernels based on image recognition according to claim 4, characterized in that, The color saturation component map and the moisture distribution map are adjusted to the same spatial resolution using a bicubic interpolation algorithm.

6. The method for identifying moldy corn kernels based on image recognition according to claim 1, characterized in that, The specific implementation method of using a pre-trained deep convolutional network to encode the mold-sensitive feature tensor to obtain multi-level feature maps is as follows: The mold-sensitive feature tensor is input into a pre-trained residual network model; The residual network model passes through multiple convolutional blocks in sequence. Each convolutional block outputs an intermediate feature map. The spatial size of the intermediate feature map decreases as the convolutional block becomes deeper, while the number of channels increases as the convolutional block becomes deeper. The intermediate feature maps output by the last three convolutional blocks in the residual network model are extracted and used as feature maps for the multiple layers.

7. The method for identifying moldy corn kernels based on image recognition according to claim 6, characterized in that, The pre-trained residual network model is a ResNet-50 model pre-trained on the ImageNet dataset.

8. The method for identifying moldy corn kernels based on image recognition according to claim 7, characterized in that, The specific implementation method for calculating the structural similarity index between adjacent hierarchical feature maps and generating a heat map of significant moldy areas based on the structural similarity index is as follows: The deepest and highest-level feature map among the multiple layers is bilinearly upsampled so that its spatial dimensions are consistent with those of the intermediate-level feature maps. Calculate the structural similarity index of each local window of the upsampled highest-layer feature map and the intermediate-layer feature map to obtain the first similarity mapping map; The intermediate layer feature map is then bilinearly upsampled to make its spatial size consistent with that of the shallowest layer feature map, and the structural similarity index between the upsampled intermediate layer feature map and the shallow layer feature map is calculated to obtain the second similarity mapping map. The heat map of the significantly moldy area is obtained by averaging the first similarity map and the second similarity map by pixels.

9. The method for identifying moldy corn kernels based on image recognition according to claim 1, characterized in that, The specific implementation method for obtaining the initial moldy region mask by adaptive threshold segmentation of the heat map is as follows: Calculate the intensity values ​​of all pixels in the heatmap and generate an intensity histogram; The optimal segmentation threshold for the intensity histogram is calculated using the Otsu method. Pixels with an intensity value greater than the optimal segmentation threshold in the heat map are marked as suspected moldy pixels, and pixels with an intensity value less than or equal to the optimal segmentation threshold are marked as background pixels. Connectivity analysis is performed on the marked suspected moldy pixels to remove isolated connected components with an area smaller than the preset minimum moldy area, thus obtaining the initial moldy region mask.

Images