An image recognition-based determination method for herbal fermentation degree

Abstract

This invention discloses a method for determining the fermentation degree of herbal medicine based on image recognition, comprising: generating corrected fermentation time-series image data of herbal medicine; obtaining HSV color feature voxel data; outputting mask-corrected HSV color feature voxel data; forming time-series visual feature data; constructing a Gaussian gradient dynamic model with fermentation degree as the latent state variable; performing Bayesian online inference on the time-series visual feature data based on the Gaussian gradient dynamic model to generate calibrated fermentation degree estimation data; and generating a fermentation degree curve in real time based on the calibrated fermentation degree estimation data and estimating the remaining time to reach the target fermentation degree. This invention can quickly and accurately adaptively reset and predict the fermentation state under multi-event intervention conditions, avoiding the defects of overfitting in the early stage and lag in the later stage, thus greatly improving the timeliness and reliability of fermentation degree determination and process control.

Description

Technical Field

[0001] This invention relates to the field of herbal medicine technology, and in particular to a method for determining the degree of fermentation of herbs based on image recognition. Background Technology

[0002] With the increasing demand for product quality control and automated management in the modern Chinese medicine manufacturing industry, online monitoring and intelligent judgment of herbal fermentation processes have become an important research direction in the field of biopharmaceutical engineering.

[0003] Currently, the determination of the fermentation degree of herbal medicines mainly relies on manual sampling, physicochemical analysis, and subjective visual interpretation, resulting in low efficiency, slow response, significant subjectivity, and batch-to-batch incomparability. In recent years, some studies have proposed using machine vision and image processing methods to analyze changes in the color and texture of fermented herbs, thereby assisting in the determination of the fermentation process.

[0004] However, existing image analysis-based fermentation degree determination methods mostly use RGB color space or simple image statistical features as quality characterization means. They are limited by the influence of RGB on various factors such as ambient light, humidity, high light, differences in camera equipment, and obstruction by herb accumulation, making it difficult to achieve stable and reliable quality evaluation across multiple batches and scenarios.

[0005] Furthermore, traditional methods often employ static regression or fixed-structure time series modeling to address the dynamic process of fermentation degree evolution over time. These methods are insufficient to effectively describe the non-stationary dynamic characteristics of the coexistence of "slow change" and "abrupt change" during fermentation. Existing Gaussian process modeling or Kalman filtering methods can only smoothly track the stationary phase of fermentation and are insufficient in responding to sudden changes driven by process events such as tank turning and feeding, which can easily lead to lag estimation or abnormal misjudgment. Summary of the Invention

[0006] One objective of this invention is to propose an image recognition-based method for determining the fermentation degree of herbs. Under multi-event intervention conditions, this invention can quickly and accurately adaptively reset and predict the fermentation state, avoiding the defects of overfitting in the early stage and lag in the later stage, thus greatly improving the timeliness and reliability of fermentation degree determination and process control.

[0007] A method for determining the degree of fermentation of herbs based on image recognition according to an embodiment of the present invention includes:

[0008] Collect time-series image data of herbal fermentation process and preprocess it to generate corrected time-series image data of herbal fermentation.

[0009] HSV color feature reconstruction was performed on the corrected herbal fermentation time sequence image data. Based on the backlighting model, RGB was converted to HSV channels and color gamut mapping was completed to obtain HSV color feature voxel data.

[0010] Generate color occlusion correction mask data from HSV color feature voxel data, and output the mask-corrected HSV color feature voxel data.

[0011] Pixel statistical features, texture features and temporal gradient features are extracted from the masked HSV color feature voxel data to form time-series visual feature data.

[0012] A Gaussian splashing dynamic model with fermentation degree as the hidden state variable is constructed. The time series visual feature data is used as the observation input to the Gaussian splashing dynamic model. A time-varying bandwidth gradient kernel and a process event-driven splashing kernel are introduced to construct the dynamic model structure.

[0013] Based on the Gaussian gradient dynamic model, Bayesian online inference is performed on the time series visual feature data to output online fermentation degree estimation data. Chemical reference sample data is collected and multi-source calibration is performed between the chemical reference sample data and the online fermentation degree estimation data to generate calibrated fermentation degree estimation data.

[0014] Based on the calibrated fermentation degree estimation data, a fermentation degree curve is generated in real time, and the remaining time to reach the target fermentation degree is estimated.

[0015] Optionally, the acquisition and preprocessing of herbal fermentation time-series image data includes:

[0016] A fixed light source and a visible light camera are set up at the fermentation site. Herbal fermentation time sequence image data are collected at preset time intervals. Each frame of the collected image is called an image frame, and all image frames are composed of an image sequence.

[0017] Lens distortion correction is performed on each frame of the image sequence to obtain the pixel coordinates after lens distortion correction;

[0018] Perform camera response linearization processing on each frame of the image after lens distortion correction to obtain each frame of the image under linear response;

[0019] Each frame of the image under linear response is subjected to initial white balance adjustment to obtain corrected herbal fermentation time series image data.

[0020] Optionally, the step of performing HSV color feature reconstruction on the corrected herbal fermentation time-series image data includes:

[0021] For each frame of the corrected herbal fermentation time-series image data, the pixel values ​​of the red, green, and blue channels are sequentially converted into hue, saturation, and luminance channel pixel values. The hue, saturation, and luminance channel pixel values ​​of all pixels are then used to form the hue, saturation, and luminance channel tensors.

[0022] The pixel values ​​of the luminance channel are corrected to obtain the intrinsic luminance channel pixel values. The intrinsic luminance channel values ​​of all pixels together form the intrinsic luminance channel tensor.

[0023] The hue channel tensor is processed to be continuous, mapping the hue channel pixel values ​​to two-dimensional projected coordinates on the hue circle, and each pixel in each frame of the image is processed to form a hue vector tensor.

[0024] Color space conversion, luminance channel correction, and hue channel continuity processing are performed sequentially on all image frames. The hue vector tensor, saturation channel tensor, and intrinsic luminance channel tensor of all image frames are combined to form the HSV color feature voxel dataset.

[0025] Optionally, generating color occlusion correction mask data in HSV color feature voxel data includes:

[0026] Generate a specular reflection binary mask for each pixel in the HSV color feature voxel dataset.

[0027] Perform a morphological opening operation on the specular reflection binary mask of each frame image to obtain the specular reflection opening mask.

[0028] For the intrinsic brightness channel tensor and saturation channel tensor of each frame image, at the pixel position where the specular reflection opening operation mask is one, the current pixel value is replaced by the average value of all pixels in the pixel neighborhood centered on the corresponding pixel with a radius of a specified neighborhood, thus obtaining the specular reflection corrected intrinsic brightness channel tensor and specular reflection corrected saturation channel tensor respectively.

[0029] For the intrinsic brightness channel tensor after specular reflection correction of each frame of image, calculate the gradient magnitude of each pixel, and use the brightness gradient threshold to generate an occlusion detection binary mask.

[0030] For each frame of the image, the specular reflection opening operation mask and the occlusion detection binary mask are logically ORed to obtain the color occlusion correction mask.

[0031] Apply the color occlusion correction mask to the specular reflection corrected HSV channel tensor, perform masking on the pixel positions where the color occlusion correction mask is one, and keep the pixel value unchanged for the pixel positions where the color occlusion correction mask is zero, to obtain the masked HSV color feature voxel dataset.

[0032] Optionally, the formation of time-series visual feature data includes:

[0033] For each frame of the HSV color feature voxel dataset after mask correction, calculate the mean, variance, and skewness of the pixel values, and use them as the pixel statistical features of the corresponding frame in the channel dimension.

[0034] For each frame of the HSV color feature voxel dataset after mask correction, texture features are calculated using local binary mode.

[0035] For two consecutive frames of images, calculate the average pixel value of the current frame and the previous frame in the same channel, and subtract the average pixel value of the previous frame from the average pixel value of the current frame to obtain the temporal gradient feature of the corresponding channel.

[0036] The pixel statistical features, texture features, and temporal gradient features of each frame of image are combined in chronological order to form a time-series visual feature dataset.

[0037] Optionally, the construction of the Gaussian gradual dynamic model with fermentation degree as the hidden state variable includes:

[0038] The covariance function of the Gaussian splashing dynamic model is constructed by weighted summation of the gradient kernel and the splashing kernel, which is used to describe the dynamic change characteristics of the fermentation degree over time.

[0039] Based on the Gaussian gradual dynamic model, the conditional probability of the visual observation feature vector at each time step and the fermentation degree hidden state at the current time step is modeled as a Gaussian distribution.

[0040] A joint likelihood function is constructed, and a set of parameters for the Gaussian progressive dynamic model is defined. By applying the Gaussian progressive covariance matrix, observation noise, and nonlinear mapping function under the optimal set of parameters to the relationship between visual observation feature data and fermentation degree latent state at all times, a Gaussian progressive dynamic model with fermentation degree as latent state and time series visual features as observations is formed.

[0041] Optionally, the gradient kernel is a time-varying bandwidth Gaussian kernel, and the gradient kernel value is equal to the ratio of the variance parameter of the smooth evolution of the fermentation process multiplied by the square of the time difference to the square of the time-varying bandwidth function, which is a negative exponential function of the independent variable.

[0042] Optionally, the splash kernel is defined as a process event-driven splash kernel. The splash kernel value is equal to the weighted sum of the effects of all process events. The effect of each process event is equal to the negative exponential function of the ratio of the sum of the squares of the event influence intensity multiplied by the time difference between the current time and the event occurrence time to the square of the time spread scale of the event influence. The splash kernel takes into account the effects of all process events.

[0043] Optionally, the step of collecting chemical reference sample data and performing multi-source calibration between the chemical reference sample data and the online fermentation degree estimation data includes:

[0044] The state increment variance is obtained by combining the covariance function of the Gaussian dynamic model at the current time with the covariance function of the Gaussian dynamic model at the previous time and the correlation between them. The prediction covariance at the current time is obtained by adding the prediction covariance at the previous time and the current state increment variance.

[0045] The nonlinear mapping function is linearized once at the predicted mean to obtain the Jacobian matrix;

[0046] According to the Bayesian online inference rule, the Kalman gain is calculated using the prediction covariance, which includes the state increment variance. The Kalman gain is then used to add the product of the prediction mean and the current observation minus the value of the nonlinear mapping function at the prediction mean to obtain the visual posterior mean of the fermentation degree hidden state. At the same time, the product of the Kalman gain and the Jacobian matrix is ​​subtracted from the product of the identity matrix and the prediction covariance to obtain the visual posterior covariance of the fermentation degree hidden state.

[0047] Chemical reference sample data is obtained by multiplying the product of the proportional coefficient vector of the latent state of fermentation degree and the product of the latent state of fermentation degree, the chemical reference index bias vector, and the chemical measurement noise.

[0048] For each chemical reference sample sampling time, the visual posterior mean and visual posterior covariance of the latent state of fermentation degree are used as prior information to perform chemical calibration update in combination with the chemical reference sample data. The chemical calibration Kalman gain is calculated, and the chemical calibration Kalman gain is used to calculate the fermentation degree estimate mean and fermentation degree estimate covariance. The fermentation degree estimate mean and fermentation degree estimate covariance constitute the calibrated fermentation degree estimate data.

[0049] Optionally, the step of generating a fermentation degree curve and estimating the remaining time to reach the target fermentation degree based on the calibrated fermentation degree estimation data in real time includes:

[0050] At each moment during the herbal fermentation cycle, a fermentation degree curve is constructed based on the mean fermentation degree estimate in the calibrated fermentation degree estimate data;

[0051] The confidence band of the fermentation degree curve is obtained by multiplying the mean of the fermentation degree estimate at each time step by the square root of the normal distribution quantile at the significance level and the covariance of the fermentation degree estimate, and the confidence interval of the fermentation degree at time t is obtained.

[0052] Set a target fermentation degree and calculate the target remaining difference at the current time. When the target remaining difference is less than or equal to zero, it is determined that the target fermentation degree has been reached or exceeded, and the remaining time to reach the target fermentation degree is recorded as zero.

[0053] Based on the preset time interval and the average fermentation degree estimate of two adjacent time points, calculate the fermentation degree change rate and the uncertainty of the fermentation degree change rate at the current time.

[0054] Under the condition that the target residual difference is greater than zero and the rate of change of fermentation degree is greater than zero, the remaining time estimate is calculated. The uncertainty of the remaining time estimate is obtained by using the uncertainty of the covariance of fermentation degree estimate and the rate of change of fermentation degree at the current time. The confidence interval of the remaining time estimate is calculated based on the uncertainty of the remaining time estimate at the significance level.

[0055] Align the fermentation degree curve with the confidence band of the fermentation degree curve on the time axis and output the remaining time estimate and its confidence interval, thus completing the real-time generation of the fermentation degree curve and the real-time estimation of the remaining time to reach the target fermentation degree.

[0056] The beneficial effects of this invention are:

[0057] (1) This invention improves the stability and cross-scene adaptability of fermentation degree measurement by deeply combining Gaussian gradient dynamic modeling and HSV color feature reconstruction. By establishing an HSV color feature voxel data structure based on backlight correction and specular reflection suppression, non-chemical deviations caused by environment and equipment are explicitly separated. Combined with hue continuity mapping and occlusion correction, robust quantification of color and texture in the herbal fermentation process is achieved. The processed HSV color features are used as the observation input to the Gaussian gradient dynamic model, which fully integrates the two types of time behavior: process events and slow ripening, and realizes continuous, stable and cross-equipment comparable measurement of herbal fermentation degree.

[0058] (2) By introducing a time-varying bandwidth gradient kernel and a process event-driven splash kernel, this invention achieves a unified modeling of the dual behaviors of stable ripening and sudden transition in the fermentation process. The gradient kernel can dynamically adjust the smoothing scale according to the fermentation cycle, suppress early noise, and accurately track late mutations. The splash kernel effectively captures state transitions by utilizing the impulse response of process events. Under multi-event intervention conditions, it can quickly and accurately adaptively reset and predict the fermentation state, avoiding the defects of early overfitting and late lag, thus greatly improving the timeliness and reliability of fermentation degree measurement and process control.

[0059] (3) By integrating small sample chemical reference data and visual estimation results, this invention uses a Bayesian multi-source calibration mechanism to jointly optimize the mean and covariance of fermentation degree estimation. It can dynamically output the confidence interval of fermentation degree estimation during real-time monitoring, support the on-site setting of fermentation targets and alarm thresholds, and improve the decision accuracy of automatic release and anomaly detection. Attached Figure Description

[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0061] Figure 1 This is a flowchart of a method for determining the degree of fermentation of herbs based on image recognition proposed in this invention;

[0062] Figure 2 This invention relates to a Gaussian gradient dynamic modeling structure for a herbal fermentation degree determination method based on image recognition. Detailed Implementation

[0063] Example 1: Reference Figures 1-2 A method for determining the degree of fermentation of herbs based on image recognition, comprising:

[0064] Collect time-series image data of herbal fermentation process and preprocess it to generate corrected time-series image data of herbal fermentation.

[0065] In this embodiment, the herbal fermentation time-series image data of the herbal fermentation process is collected and preprocessed, including:

[0066] A fixed light source and a visible light camera are set up at the fermentation site. Herbal fermentation time sequence image data are collected at preset time intervals. Each frame of the collected image is called an image frame, and all image frames are composed of an image sequence.

[0067] Each frame in the image sequence is numbered according to its acquisition time, and the length of the image sequence is equal to the total number of frames acquired during the fermentation cycle.

[0068] Lens distortion correction is performed on each frame of the image sequence to obtain the pixel coordinates after lens distortion correction;

[0069] Lens distortion correction maps the original coordinates of each pixel to the radial and tangential distortion parameters of the camera to obtain the pixel coordinates after lens distortion correction. All pixel coordinates are corrected based on the parameters obtained in advance in the equipment calibration experiment from the calibration board image.

[0070] Perform camera response linearization processing on each frame of the image after lens distortion correction to obtain each frame of the image under linear response;

[0071] Camera response linearization is achieved by applying the inverse function of the camera response function to the original pixel values ​​of the red, green, and blue channels respectively, restoring the original pixel values ​​to pixel values ​​under linear response. The inverse functions of the camera response for all channels were obtained through equipment calibration experiments.

[0072] Each frame of the image under linear response is subjected to initial white balance adjustment to obtain corrected herbal fermentation time series image data.

[0073] The initial white balance adjustment process involves selecting a reference gray area in the image, calculating the average brightness values ​​of the red, green, and blue channels within that area, dividing the average brightness value by the target brightness value to obtain the channel gain coefficient, and multiplying the linear pixel value of each channel by the corresponding channel gain coefficient to obtain the image after initial white balance adjustment. The image after initial white balance adjustment serves as corrected herbal fermentation time-series image data with lens distortion corrected, response linearized, and white balance initially adjusted.

[0074] HSV color feature reconstruction was performed on the corrected herbal fermentation time sequence image data. Based on the backlighting model, RGB was converted to HSV channels and color gamut mapping was completed to obtain HSV color feature voxel data.

[0075] In this embodiment, HSV color feature reconstruction is performed on the corrected herbal fermentation time-series image data, including:

[0076] For each frame of the corrected herbal fermentation time-series image data, the pixel values ​​of the red, green, and blue channels are sequentially converted into hue, saturation, and luminance channel pixel values. The hue, saturation, and luminance channel pixel values ​​of all pixels are then used to form the hue, saturation, and luminance channel tensors.

[0077] The pixel value of the luminance channel is equal to the pixel value of the channel with the largest value among the red, green, and blue channels.

[0078] The pixel value of the saturation channel is equal to the difference between the pixel value of the luminance channel and the pixel value of the smallest of the three channels (red, green, and blue), divided by the pixel value of the luminance channel.

[0079] The pixel values ​​of the hue channel are calculated according to the following rules, depending on whether the pixel values ​​of the luminance channel correspond to the red, green, or blue channels:

[0080] When the pixel value of the luminance channel is equal to the pixel value of the red channel and the pixel value of the green channel is greater than or equal to the pixel value of the blue channel, the pixel value of the hue channel is equal to 60 degrees multiplied by the difference between the pixel values ​​of the green and blue channels, and then divided by the difference between the pixel value of the luminance channel and the pixel value of the channel with the smallest value among the red, green, and blue channels.

[0081] When the pixel value of the luminance channel is equal to the pixel value of the red channel and the pixel value of the green channel is less than the pixel value of the blue channel, the pixel value of the hue channel is equal to 60 degrees multiplied by the difference between the pixel values ​​of the green channel and the blue channel, divided by the difference between the pixel value of the luminance channel and the pixel value of the smallest of the three channels (red, green, and blue), plus 360 degrees.

[0082] When the pixel value of the luminance channel equals the pixel value of the green channel, the pixel value of the hue channel equals 60 degrees multiplied by the difference between the pixel values ​​of the blue and red channels, divided by the difference between the pixel value of the luminance channel and the pixel value of the smallest of the three channels (red, green, and blue), plus 120 degrees.

[0083] When the pixel value of the luminance channel equals the pixel value of the blue channel, the pixel value of the hue channel equals 60 degrees multiplied by the difference between the pixel values ​​of the red and green channels, divided by the difference between the pixel value of the luminance channel and the pixel value of the smallest of the three channels (red, green, and blue), plus 240 degrees.

[0084] The pixel values ​​of the luminance channel are corrected to obtain the intrinsic luminance channel pixel values. The intrinsic luminance channel values ​​of all pixels together form the intrinsic luminance channel tensor.

[0085] The correction method is to convert the pixel value of the brightness channel of each pixel into logarithmic form. The logarithmic value is equal to the sum of the logarithmic value of the inherent brightness channel pixel value and the logarithmic value of the incident light intensity. The logarithmic value of the incident light intensity of each pixel is estimated and then subtracted from the logarithmic value of the observed brightness channel pixel value to obtain the logarithmic value of the inherent brightness channel pixel value. Finally, the logarithmic value of the inherent brightness channel pixel value is restored by exponential transformation to obtain the inherent brightness channel pixel value.

[0086] The hue channel tensor is processed to be continuous, mapping the hue channel pixel values ​​to two-dimensional projected coordinates on the hue circle, and each pixel in each frame of the image is processed to form a hue vector tensor.

[0087] The continuous processing method involves dividing the hue channel pixel value by 360 degrees for each pixel position, and then calculating the cosine and sine values ​​to obtain the two-dimensional projection coordinates of the corresponding pixel on the hue circle.

[0088] Color space conversion, luminance channel correction, and hue channel continuity processing are performed sequentially on all image frames. The hue vector tensor, saturation channel tensor, and intrinsic luminance channel tensor of all image frames are combined to form the HSV color feature voxel dataset.

[0089] Generate color occlusion correction mask data from HSV color feature voxel data, and output the mask-corrected HSV color feature voxel data.

[0090] In this embodiment, color occlusion correction mask data is generated from HSV color feature voxel data, including:

[0091] Generate a specular reflection binary mask for each pixel in the HSV color feature voxel dataset.

[0092] The generation rules for specular reflection binary masks are as follows:

[0093] When the pixel value of a pixel in the saturation channel tensor is less than the preset saturation threshold and the pixel value in the intrinsic brightness channel tensor is greater than the preset brightness threshold, the specular reflection binary mask value corresponding to the pixel is set to one; otherwise, it is set to zero.

[0094] Perform a morphological opening operation on the specular reflection binary mask of each frame image to obtain the specular reflection opening mask.

[0095] For the intrinsic brightness channel tensor and saturation channel tensor of each frame image, at the pixel position where the specular reflection opening operation mask is one, the current pixel value is replaced by the average value of all pixels in the pixel neighborhood centered on the corresponding pixel with a radius of a specified neighborhood, thus obtaining the specular reflection corrected intrinsic brightness channel tensor and specular reflection corrected saturation channel tensor respectively.

[0096] For the intrinsic brightness channel tensor after specular reflection correction of each frame of image, calculate the gradient magnitude of each pixel, and use the brightness gradient threshold to generate an occlusion detection binary mask.

[0097] If the gradient magnitude of a pixel is less than the brightness gradient threshold, the corresponding pixel's occlusion detection binary mask value is set to one; otherwise, it is set to zero. The brightness gradient threshold is set with uniform dimensions in the herbal fermentation scenario.

[0098] For each frame of the image, the specular reflection opening operation mask and the occlusion detection binary mask are logically ORed to obtain the color occlusion correction mask.

[0099] Apply the color occlusion correction mask to the specular reflection corrected HSV channel tensor, perform masking on the pixel positions where the color occlusion correction mask is one, and keep the pixel value unchanged for the pixel positions where the color occlusion correction mask is zero, to obtain the masked HSV color feature voxel dataset.

[0100] The specular-corrected HSV channel tensor consists of a hue vector tensor, a specular-corrected intrinsic brightness channel tensor, and a specular-corrected saturation channel tensor.

[0101] ;

[0102] in, This represents element-wise multiplication. Correct the mask for color occlusion. Let hue vector tensor be the hue vector tensor. This is the intrinsic brightness channel tensor after specular reflection correction. is the saturation channel tensor after specular reflection correction, and T is the total number of frames acquired during the herbal fermentation cycle.

[0103] Pixel statistical features, texture features and temporal gradient features are extracted from the masked HSV color feature voxel data to form time-series visual feature data.

[0104] In this embodiment, the formation of time-series visual feature data includes:

[0105] For each frame of the HSV color feature voxel dataset after mask correction, calculate the mean, variance, and skewness of the pixel values, and use them as the pixel statistical features of the corresponding frame in the channel dimension.

[0106] In Example 1, for each frame of the HSV color feature voxel dataset after mask correction, for the hue vector tensor, the saturation channel tensor after specular correction, and the intrinsic brightness channel tensor after specular correction, all pixels are traversed, and the pixel values ​​of all pixels are summed and divided by the total number of pixels to obtain the mean value of the pixel value. The mean value of each pixel value is subtracted from the mean value and then squared. All squared values ​​are summed and divided by the total number of pixels minus one to obtain the variance of the pixel value. The mean value of the pixel value reflects the overall brightness or color level of the current frame channel, the variance of the pixel value reflects the dispersion of the pixel values ​​of the current frame channel, and the skewness of the pixel value is used to reflect the symmetry of the distribution of the pixel values ​​of the current frame channel.

[0107] For each frame of the HSV color feature voxel dataset after mask correction, texture features are calculated using local binary mode.

[0108] In Example 1, the local binary mode is as follows: each pixel of each frame image is taken as the center point, and a specified number of neighboring pixels are sampled with a preset radius. The pixel value of each neighboring pixel is compared with the center pixel value. When the pixel value of the neighboring pixel is greater than or equal to the center pixel value, it is recorded as one; otherwise, it is recorded as zero. All neighbor comparison results are arranged in order to form a binary code. The decimal value of the binary code is the local binary mode code of the corresponding pixel. The histogram of the local binary mode codes of all pixels in the entire frame image is used as the texture feature of the corresponding frame.

[0109] For two consecutive frames of images, calculate the average pixel value of the current frame and the previous frame in the same channel, and subtract the average pixel value of the previous frame from the average pixel value of the current frame to obtain the temporal gradient feature of the corresponding channel.

[0110] The pixel statistical features, texture features, and temporal gradient features of each frame of image are combined in chronological order to form a time-series visual feature dataset.

[0111] S5. Construct a Gaussian splashing dynamic model with fermentation degree as the hidden state variable, take time series visual feature data as the observation input to the Gaussian splashing dynamic model, and introduce time-varying bandwidth gradient kernel and process event-driven splashing kernel to construct the dynamic model structure.

[0112] In this embodiment, a Gaussian gradual dynamic model with fermentation degree as the hidden state variable is constructed, including:

[0113] The covariance function of the Gaussian splashing dynamic model is constructed by weighted summation of the gradient kernel and the splashing kernel, which is used to describe the dynamic change characteristics of the fermentation degree over time.

[0114] The covariance function of the Gaussian progressive dynamic model is the neural center of the model, characterizing the correlation between the latent states of fermentation degree at any two time points.

[0115] The gradient kernel is a time-varying bandwidth Gaussian kernel. The value of the gradient kernel is equal to the ratio of the variance parameter of the smooth evolution of the fermentation process multiplied by the square of the time difference to the square of the time-varying bandwidth function, which is a negative exponential function of the independent variable.

[0116] The value of the time-varying bandwidth function is equal to the exponential function of the sum of the initial bandwidth multiplied by a negative bandwidth attenuation coefficient and the current moment, divided by the total number of fermentation cycle frames.

[0117] ;

[0118] ;

[0119] in, This indicates that during the herbal fermentation cycle, the first... Frame and the Between frames, the covariance of the fermentation degree hidden state under a gradient background is represented by the gradient kernel. The amplitude parameter representing the Gaussian gradient kernel has the physical meaning of the variance of the slow evolution of the degree of fermentation during the fermentation process of herbs, reflecting the overall amplitude of the degree of fermentation under the background of natural fluctuations. These represent the [number]th ... Frame and the The frame's time number has a value range of 100. , Indicates the Gaussian gradient kernel at the th Frame and the The time-varying bandwidth function between frames, The initial bandwidth of the Gaussian gradient kernel determines the range of covariance variation between hidden states during the early stages of fermentation. Indicates the bandwidth attenuation coefficient. This represents the absolute value of the sum of the time numbers of two frames, reflecting the time accumulation factor used for bandwidth attenuation. This indicates the total number of image frames acquired during the herbal fermentation cycle.

[0120] In Example 1, the gradient kernel structure using a time-varying bandwidth Gaussian kernel can adaptively adjust its smoothness according to different stages of the fermentation process, fully reflecting the dynamic essence of slow ripening in the early stage and accelerated changes in the later stage of herbal fermentation. The gradient kernel structure automatically assigns a large kernel bandwidth in the early stage of fermentation, realizing effective overall modeling of the slow evolution process and effectively suppressing the influence of noise and misjudgment of small fluctuations. In the later stage of fermentation, the bandwidth gradually narrows, which can keenly capture the dynamic process of accelerated fermentation and sharper ripening signal in the later stage. This improves the adaptability and resolution of the Gaussian gradient dynamic model to the real process state. Compared with the traditional fixed bandwidth kernel, it avoids the problem of overfitting in the early stage and lagging response in the later stage caused by using a uniform smoothing scale for the entire fermentation process, and significantly improves the accuracy and dynamic response capability of continuous estimation of herbal fermentation degree.

[0121] The splash kernel is defined as a process event-driven splash kernel. The splash kernel value is equal to the weighted sum of the effects of all process events. The effect of each process event is equal to the negative exponential function of the sum of the sum of the squares of the event effect intensity multiplied by the time difference between the current time and the event occurrence time and the square of the time spread scale of the event effect. The splash kernel takes into account the effects of all process events.

[0122] ;

[0123] in, This represents the total number of all critical process events recorded during the herbal fermentation cycle. Each process event refers to a single operation during fermentation, whether manual or automatic, that significantly alters the fermentation dynamics, such as turning, adding, or aerating the material. Each event is individually numbered. Indicates the first The data acquisition time when a process event occurs. Indicates the first The intensity of the impact of each process event on the dynamic changes of fermentation is a non-negative real-valued parameter used to quantitatively describe the impulse excitation amplitude of each process event on the latent state of fermentation. It is obtained through empirical settings. The time-scale of the impact of a process event is a positive real parameter that controls the spread width of the splash nucleus on the time axis and determines the range of influence of each process event on the covariance of fermentation degree before and after the event.

[0124] In Example 1, the introduced process event-driven splash kernel enhances the dynamic modeling capability of herbal fermentation degree determination. The splash kernel structure can represent the instantaneous impact of manual or automatic material turning, feeding, and aeration processes on the fermentation process, enabling synchronous perception of slow changes and abrupt changes in fermentation state. By superimposing the impact of each event into the covariance structure in the form of a pulsed Gaussian kernel, the Gaussian splash dynamic model can smoothly track the long-term evolution trend of fermentation degree while sensitively responding to and quantifying state transitions brought about by process operations. This avoids the lag and overfitting problems of traditional stationary kernels for abrupt signals. The introduction of the splash kernel enables the Gaussian splash dynamic model to quickly reset the state estimate when process disturbances occur, improving the real-time performance and accuracy of anomaly detection and decision support, and enhancing the adaptability and reliability of the determination method to actual complex fermentation scenarios.

[0125] Based on the Gaussian gradual dynamic model, the conditional probability of the visual observation feature vector at each time step and the fermentation degree hidden state at the current time step is modeled as a Gaussian distribution.

[0126] In Example 1, the time-series visual feature dataset is used as the observation input to the Gaussian gradient dynamic model. The fermentation degree is set as the fermentation degree hidden state that evolves over time. Each observation is the visual feature vector corresponding to a time. The visual feature vector is obtained by combining statistical features, texture features and temporal gradient features from the masked HSV color feature voxel data. The fermentation degree is set as the hidden state variable that evolves over time. A unique fermentation degree hidden state is set for each sampling time.

[0127] The value of the fermentation degree hidden state at adjacent time steps is equal to the sum of the fermentation degree hidden state value at the previous time step and the state transition noise value; the visual observation feature vector corresponding to the current time step is equal to the sum of the fermentation degree hidden state value after passing through the nonlinear mapping function and the observation noise value.

[0128] The Gaussian splashing dynamic model refers to a statistical dynamic model that weights and integrates the gradual part of the traditional Gaussian process and the splashing part caused by process events in the same kernel function and state space structure in the dynamic modeling of the non-stationary complex process of herbal fermentation.

[0129] A joint likelihood function is constructed, and a set of parameters for the Gaussian progressive dynamic model is defined. By applying the Gaussian progressive covariance matrix, observation noise, and nonlinear mapping function under the optimal set of parameters to the relationship between visual observation feature data and fermentation degree latent state at all times, a Gaussian progressive dynamic model with fermentation degree as latent state and time series visual features as observations is formed.

[0130] In Example 1, the visual observation feature data at all times are jointly modeled with the corresponding fermentation degree hidden states through probabilistic relationships. The value of the joint likelihood function is equal to the product of the probability density functions of the observation feature data at all times, the fermentation degree hidden states at all times, and the joint distribution of all fermentation degree hidden states, under the current Gaussian asymptotic covariance matrix parameters and observation noise parameters.

[0131] The parameter set of the Gaussian splashing gradual dynamic model includes the gradient kernel weighting coefficient, the splashing kernel weighting coefficient, the variance parameter of the smooth evolution of the fermentation process, the initial bandwidth, the bandwidth decay coefficient, the temporal diffusion scale of the event influence, the influence intensity of all process events, and the occurrence time of all process events. Each parameter in the parameter set of the Gaussian splashing gradual dynamic model is estimated by maximizing the joint likelihood function to obtain the optimal Gaussian splashing gradual dynamic model structure.

[0132] Based on the Gaussian gradient dynamic model, Bayesian online inference is performed on time series visual feature data to output online fermentation degree estimation data. Chemical reference sample data is collected and multi-source calibration is performed on the chemical reference sample data and the online fermentation degree estimation data to generate calibrated fermentation degree estimation data.

[0133] In this embodiment, chemical reference sample data is collected and multi-source calibration is performed on the chemical reference sample data and the online fermentation degree estimation data, including:

[0134] The state increment variance is obtained by combining the covariance function of the Gaussian dynamic model at the current time with the covariance function of the Gaussian dynamic model at the previous time and the correlation between them. The prediction covariance at the current time is obtained by adding the prediction covariance at the previous time and the current state increment variance.

[0135] ;

[0136] ;

[0137] in, Let the covariance function be the asymptotic dynamic model of Gaussian. The predictive covariance of the latent state of fermentation degree. Let V be the state increment variance.

[0138] The nonlinear mapping function is linearized once at the predicted mean to obtain the Jacobian matrix;

[0139] According to the Bayesian online inference rule, the Kalman gain is calculated using the prediction covariance, which includes the state increment variance. The Kalman gain is then used to add the product of the prediction mean and the current observation minus the value of the nonlinear mapping function at the prediction mean to obtain the visual posterior mean of the fermentation degree hidden state. At the same time, the product of the Kalman gain and the Jacobian matrix is ​​subtracted from the product of the identity matrix and the prediction covariance to obtain the visual posterior covariance of the fermentation degree hidden state.

[0140] The Kalman gain equals the product of the prediction covariance and the transpose of the Jacobian matrix, multiplied by the Jacobian matrix and the product of the prediction covariance and the transpose of the Jacobian matrix, plus the inverse product of the sum of the observation noise covariance.

[0141] ;

[0142] in, For a moment The Kalman gain in the Bayesian filtering inference of herbal fermentation degree measures the weight of incorporating time-series visual features into the latent state estimation of fermentation degree at this time step. To be within the known cutoff time Under all observations at all times, for the first The fermentation process is in a hidden state at any given time. The prior covariance, For the first The Jacobian matrix at time step is used to map the fermentation degree latent state variable to the time series visual feature space. Jacobian matrix transpose, For the first The observation noise covariance at a given time moment represents the covariance structure of random perturbations and unexplainable components in the visual features of the time series at that moment.

[0143] Chemical reference sample data is obtained by multiplying the product of the proportional coefficient vector of the latent state of fermentation degree and the product of the latent state of fermentation degree, the chemical reference index bias vector, and the chemical measurement noise.

[0144] In Example 1, the herbal fermentation sample was tested using physicochemical detection methods within the chemical reference sample sampling time set, and the actual observed values ​​of multiple chemical reference indicators were obtained. All the observed values ​​constituted the chemical reference sample data. The observation relationship was used to calibrate and fuse the latent state of fermentation degree during the multi-source calibration process.

[0145] For each chemical reference sample sampling time, the visual posterior mean and visual posterior covariance of the fermentation degree latent state are used as prior information to perform chemical calibration update, calculate the chemical calibration Kalman gain, and use the chemical calibration Kalman gain to calculate the fermentation degree estimate mean and fermentation degree estimate covariance. The fermentation degree estimate mean and fermentation degree estimate covariance constitute the calibrated fermentation degree estimate data.

[0146] In Example 1, the chemical calibration Kalman gain is equal to the product of the visual posterior covariance and the transpose of the proportionality coefficient vector of the chemical reference index to the latent state of fermentation, multiplied by the proportionality coefficient vector and the transpose of the visual posterior covariance and the proportionality coefficient vector, plus the inverse multiplication of the sum of the chemical measurement noise covariance matrix.

[0147] ;

[0148] in, This indicates the first step in the herbal fermentation process. At each sampling time point, the chemical calibration Kalman gain calculated based on the multi-source calibration mechanism is used to fuse visual estimation and chemical reference samples to achieve optimal calibration of the latent state of fermentation degree. Indicates the first The visual posterior covariance of the fermentation degree latent state inferred from the visual observation feature sequence and Gaussian gradient dynamics model at each sampling time point. This represents the posterior covariance of the observation noise of the chemical reference sample.

[0149] The chemical reference sample data is subtracted by the sum of the visual posterior mean and the bias vector, multiplied by the chemically calibrated Kalman gain, and then the visual posterior mean is added to obtain the calibrated mean of fermentation degree estimation.

[0150] ;

[0151] Where b is the chemical reference index bias vector.

[0152] The calibrated fermentation degree estimate covariance is obtained by subtracting the product of the chemically calibrated Kalman gain and the scaling factor vector from the identity matrix and then multiplying it with the visual posterior covariance.

[0153] This implementation method achieves multi-source calibration for fermentation degree estimation by dynamically fusing visual and physicochemical data, which significantly improves the accuracy and reliability of fermentation degree estimation. It adopts recursive filtering and chemical Kalman gain method to effectively suppress process noise and system drift, making the Gaussian dynamic model more adaptable to process fluctuations and environmental changes.

[0154] Based on the calibrated fermentation degree estimation data, a fermentation degree curve is generated in real time, and the remaining time to reach the target fermentation degree is estimated.

[0155] In this embodiment, a fermentation degree curve is generated in real time based on the calibrated fermentation degree estimation data, and the remaining time to reach the target fermentation degree is estimated, including:

[0156] At each moment during the herbal fermentation cycle, a fermentation degree curve is constructed based on the mean fermentation degree estimate in the calibrated fermentation degree estimate data;

[0157] The fermentation degree curve consists of the estimated mean of fermentation degree at each moment. The total number of frames acquired during the herbal fermentation cycle is equal to the total number of moments in the fermentation degree curve.

[0158] The confidence band of the fermentation degree curve is obtained by multiplying the mean of the fermentation degree estimate at each time step by the square root of the normal distribution quantile at the significance level and the covariance of the fermentation degree estimate, and the confidence interval of the fermentation degree at time t is obtained.

[0159] ;

[0160] in, This represents the confidence interval for the degree of fermentation. is the quantile of the normal distribution at the significance level.

[0161] Set a target fermentation degree and calculate the target remaining difference at the current time. When the target remaining difference is less than or equal to zero, it is determined that the target fermentation degree has been reached or exceeded, and the remaining time to reach the target fermentation degree is recorded as zero.

[0162] The target residual difference equals the target fermentation degree minus the current estimated mean fermentation degree.

[0163] Based on the preset time interval and the estimated average fermentation degree of two adjacent time points, calculate the rate of change of fermentation degree at the current time and the uncertainty of the rate of change of fermentation degree.

[0164] The rate of change of fermentation degree is obtained by subtracting the mean of the fermentation degree estimate from the mean of the fermentation degree estimate at the current time from the mean of the fermentation degree estimate at the previous time, and then dividing by the sampling time interval. The uncertainty of the rate of change of fermentation degree is obtained by dividing the sum of the covariances of the calibrated fermentation degree estimates at two adjacent time points by the square of the sampling time interval.

[0165] Under the condition that the target residual difference is greater than zero and the rate of change of fermentation degree is greater than zero, the remaining time estimate is calculated. The uncertainty of the remaining time estimate is obtained by using the uncertainty of the covariance of fermentation degree estimate and the rate of change of fermentation degree at the current time. The confidence interval of the remaining time estimate is calculated based on the uncertainty of the remaining time estimate at the significance level.

[0166] The remaining time estimate equals the target remaining difference divided by the rate of change of fermentation degree.

[0167] The upper and lower limits of the confidence interval for the remaining time estimate are obtained by multiplying the quantiles of the normal distribution at the significance level plus or minus the remaining time estimate by the uncertainty of the remaining time estimate.

[0168] ;

[0169] in, Estimate the uncertainty for the remaining time. The rate of change of fermentation degree, The target remaining difference, This represents the uncertainty of the rate of change of fermentation degree.

[0170] Align the fermentation degree curve with the confidence band of the fermentation degree curve on the time axis and output the remaining time estimate and its confidence interval, thus completing the real-time generation of the fermentation degree curve and the real-time estimation of the remaining time to reach the target fermentation degree.

[0171] This implementation method helps operators intuitively control the fermentation process by visualizing the fermentation degree trend and quantifying the uncertainty range, thereby improving the scientific nature and foresight of process management. The intelligent prediction of remaining time significantly improves the efficiency of process scheduling, helps to accurately control the fermentation window, reduces human judgment errors, and enhances the controllability and automation level of process quality.

[0172] Example 2: On a herbal fermentation production line, the fermentation process of a batch of newly arrived herbal raw materials was monitored and measured automatically and continuously using the method of this invention. To compare and verify the effect, a traditional RGB visual method combined with static regression was used concurrently (control group). The entire process is as follows:

[0173] The system first set up two calibrated visible light cameras and a standard light source in the fermentation zone, and set the sampling frequency to 1 frame every 20 minutes. A total of 720 frames of images were collected during the entire fermentation cycle.

[0174] After fermentation begins, the system automatically acquires each frame of image and performs lens distortion correction, camera response linearization, and initial white balance adjustment for gray areas in real time. Taking frame 48 as an example, before correction, the edge pixels of the image had a 3.7% positional offset. After radial and tangential parameter correction, the error of all pixels was less than 0.5%. Before white balance, the average value of the red channel was 163, the green channel was 151, and the blue channel was 139. After correction, all values ​​were normalized to the target brightness of 145±2.

[0175] For each pixel in the corrected image, the system sequentially calculates the hue, saturation, and brightness channels. In Example 2, the RGB values ​​of a certain region in frame 48 are (172, 159, 141). After conversion, the hue channel is 33°, the saturation is 0.18, and the brightness channel is 0.67. All pixels in the frame undergo HSV conversion, forming a three-channel tensor.

[0176] In the 62nd frame of the acquisition, a wet highlight appeared on the surface of the fermentation bed. 2185 pixels were detected in the saturation channel with values ​​below the threshold of 0.10 and in the brightness channel above 0.85, and were automatically identified as specular reflection areas. After masking, the pixel values ​​of these areas were replaced with the average of their surrounding 3 pixels. Furthermore, 832 occlusion points caused by clumping were identified using a brightness gradient threshold; these were also corrected using a mask, and the resulting HSV feature voxels were output after mask correction.

[0177] Taking frame 100 as an example, the hue channel has a mean of 31.2, a variance of 2.6, and a skewness of 0.08. In the LBP histogram of the saturation channel, the "Type 1" mode accounts for 42.3%, and the "Type 5" mode accounts for 27.7%. The mean brightness change between two consecutive frames is 0.008. The system stores the above statistical features, LBP texture features, and temporal gradients frame by frame, forming a time-series visual feature set of length 720.

[0178] The time-series visual feature set was input into the Gaussian splashing dynamic model. The initial settings included a gradient kernel bandwidth (l0) of 2.5 days and a bandwidth decay coefficient (β) of 0.32. On fermentation days 4, 8, and 11, material turning, material replenishment, and aeration events occurred, respectively. The system automatically recorded the acquisition times of these events as 192, 384, and 528. The system used the feature vector at each time point as an observation and constructed a weighted covariance matrix between time points using the gradient kernel and the splashing kernel.

[0179] At frame 300 (day 8), manual sampling measured the pH at 4.18 and the reducing sugar at 0.71 g / 100g. The system added the predicted covariance of the previous 299 frames to the current incremental variance, calculating a state prediction covariance of 0.025. With the features of the current frame as input and a Kalman gain of 0.62, the system fused the visual estimate mean of 3.91 and the chemical observation value of 4.18. After calibration, the fermentation degree mean was 4.07, and the covariance decreased to 0.015. Data fusion was completed for 30 chemical calibration points throughout the cycle.

[0180] The system plots the calibrated average fermentation degree as a fermentation degree curve in real time. At frame 500, the average fermentation degree is 0.89, and the covariance is 0.009. The 95% confidence interval is [0.870, 0.910]. The target fermentation degree is set at 0.92. Based on the current rate of change of 0.007 / frame and the target remaining difference of 0.03, the system automatically estimates the remaining time as 4.3 hours, with a confidence interval width of 0.8 hours, and prompts "The fermentation target is expected to be reached within this shift. Please prepare for release."

[0181] During this batch process, the same batch of data was simultaneously processed using the traditional RGB mean-variance + fixed bandwidth kernel model. The comparison results are shown in Table 1 below:

[0182] Table 1. Comparison of key data points between the present invention and the traditional method.

[0183]

[0184] At sudden events such as material turning / ventilation, the response time lag of the traditional method is about 8 frames (2.7 hours), while the response time lag of the model of this invention is reduced to less than 1.5 frames (0.5 hours) under the action of splash nuclei. Over the entire cycle, the mean square error of fermentation degree estimation of the traditional method is 0.096, while that of the method of this invention is 0.013, and the maximum confidence interval width of the cycle is 0.66 times that of the traditional method.

[0185] During all 12 fermentation processes, the average batch-to-batch error of the traditional method was 0.089, while that of the present invention was 0.022. The accuracy of the release point was increased from 82% of the traditional method to 99% of the present invention. In 6 batches with high humidity and drastic light changes, the traditional method had 3 false alarms or missed alarms, while the present invention had no missed alarms. All batches could automatically alarm when there were process abnormalities.

[0186] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for determining the degree of fermentation of herbs based on image recognition, characterized in that, include: Collect time-series image data of herbal fermentation process and preprocess it to generate corrected time-series image data of herbal fermentation. HSV color feature reconstruction was performed on the corrected herbal fermentation time sequence image data. Based on the backlighting model, RGB was converted to HSV channels and color gamut mapping was completed to obtain HSV color feature voxel data. Generate color occlusion correction mask data from HSV color feature voxel data, and output the mask-corrected HSV color feature voxel data. Pixel statistical features, texture features and temporal gradient features are extracted from the masked HSV color feature voxel data to form time-series visual feature data. A Gaussian splashing dynamic model with fermentation degree as the hidden state variable is constructed. The time series visual feature data is used as the observation input to the Gaussian splashing dynamic model. A time-varying bandwidth gradient kernel and a process event-driven splashing kernel are introduced to construct the dynamic model structure. Based on the Gaussian gradient dynamic model, Bayesian online inference is performed on the time series visual feature data to output online fermentation degree estimation data. Chemical reference sample data is collected and multi-source calibration is performed between the chemical reference sample data and the online fermentation degree estimation data to generate calibrated fermentation degree estimation data. Based on the calibrated fermentation degree estimation data, a fermentation degree curve is generated in real time, and the remaining time to reach the target fermentation degree is estimated.

2. The method for determining the degree of fermentation of herbs based on image recognition according to claim 1, characterized in that, The process of collecting and preprocessing time-series image data of herbal fermentation includes: A fixed light source and a visible light camera are set up at the fermentation site. Herbal fermentation time sequence image data are collected at preset time intervals. Each frame of the collected image is called an image frame, and all image frames form an image sequence. Lens distortion correction is performed on each frame of the image sequence to obtain the pixel coordinates after lens distortion correction; Perform camera response linearization processing on each frame of the image after lens distortion correction to obtain each frame of the image under linear response; Each frame of the image under linear response is subjected to initial white balance adjustment to obtain corrected herbal fermentation time series image data.

3. The method for determining the degree of fermentation of herbs based on image recognition according to claim 1, characterized in that, The process of performing HSV color feature reconstruction on the corrected herbal fermentation time-series image data includes: For each frame of the corrected herbal fermentation time-series image data, the pixel values ​​of the red, green, and blue channels are sequentially converted into hue, saturation, and luminance channel pixel values. The hue, saturation, and luminance channel pixel values ​​of all pixels are then used to form the hue, saturation, and luminance channel tensors. The pixel values ​​of the luminance channel are corrected to obtain the intrinsic luminance channel pixel values. The intrinsic luminance channel values ​​of all pixels together form the intrinsic luminance channel tensor. The hue channel tensor is processed to be continuous, mapping the hue channel pixel values ​​to two-dimensional projected coordinates on the hue circle, and each pixel in each frame of the image is processed to form a hue vector tensor. Color space conversion, luminance channel correction, and hue channel continuity processing are performed sequentially on all image frames. The hue vector tensor, saturation channel tensor, and intrinsic luminance channel tensor of all image frames are combined to form the HSV color feature voxel dataset.

4. The method for determining the degree of fermentation of herbs based on image recognition according to claim 1, characterized in that, The process of generating color occlusion correction mask data from HSV color feature voxel data includes: Generate a specular reflection binary mask for each pixel in the HSV color feature voxel dataset. Perform a morphological opening operation on the specular reflection binary mask of each frame of the image to obtain the specular reflection opening mask. For the intrinsic brightness channel tensor and saturation channel tensor of each frame image, at the pixel position where the specular reflection opening operation mask is one, the current pixel value is replaced by the average value of all pixels in the pixel neighborhood centered on the corresponding pixel with a radius of a specified neighborhood, thus obtaining the specular reflection corrected intrinsic brightness channel tensor and specular reflection corrected saturation channel tensor respectively. For the intrinsic brightness channel tensor after specular reflection correction of each frame image, calculate the gradient magnitude of each pixel, and use the brightness gradient threshold to generate an occlusion detection binary mask. For each frame of the image, the specular reflection opening operation mask and the occlusion detection binary mask are logically ORed to obtain the color occlusion correction mask. Apply the color occlusion correction mask to the specular reflection corrected HSV channel tensor, perform masking on the pixel positions where the color occlusion correction mask is one, and keep the pixel value unchanged for the pixel positions where the color occlusion correction mask is zero, to obtain the masked HSV color feature voxel dataset.

5. The method for determining the degree of fermentation of herbs based on image recognition according to claim 1, characterized in that, The formation of time-series visual feature data includes: For each frame of the HSV color feature voxel dataset after mask correction, calculate the mean, variance, and skewness of the pixel values, and use them as the pixel statistical features of the corresponding frame in the channel dimension. For each frame of the HSV color feature voxel dataset after mask correction, texture features are calculated using local binary mode. For two consecutive frames of images, calculate the average pixel value of the current frame and the previous frame in the same channel, and subtract the average pixel value of the previous frame from the average pixel value of the current frame to obtain the temporal gradient feature of the corresponding channel. The pixel statistical features, texture features, and temporal gradient features of each frame of image are combined in chronological order to form a time-series visual feature dataset.

6. The method for determining the degree of fermentation of herbs based on image recognition according to claim 1, characterized in that, The construction of the Gaussian gradual dynamic model with fermentation degree as the hidden state variable includes: The covariance function of the Gaussian splashing dynamic model is constructed by weighted summation of the gradient kernel and the splashing kernel, which is used to describe the dynamic change characteristics of the fermentation degree over time. Based on the Gaussian gradual dynamic model, the conditional probability of the visual observation feature vector at each time step and the fermentation degree hidden state at the current time step is modeled as a Gaussian distribution. A joint likelihood function is constructed, and a set of parameters for the Gaussian progressive dynamic model is defined. By applying the Gaussian progressive covariance matrix, observation noise, and nonlinear mapping function under the optimal set of parameters to the relationship between visual observation feature data and fermentation degree latent state at all times, a Gaussian progressive dynamic model with fermentation degree as latent state and time series visual features as observations is formed.

7. The method for determining the degree of fermentation of herbs based on image recognition according to claim 6, characterized in that, The gradient kernel is a time-varying bandwidth Gaussian kernel. The value of the gradient kernel is equal to the ratio of the variance parameter of the smooth evolution of the fermentation process multiplied by the square of the time difference to the square of the time-varying bandwidth function, which is a negative exponential function of the independent variable.

8. The method for determining the degree of fermentation of herbs based on image recognition according to claim 6, characterized in that, The splash kernel is defined as a process event-driven splash kernel. The splash kernel value is equal to the weighted sum of the effects of all process events. The effect of each process event is equal to the negative exponential function of the ratio of the sum of the squares of the event influence intensity multiplied by the sum of the squares of the time difference between the current time and the event occurrence time and the square of the time spread scale of the event influence. The splash kernel takes into account the effects of all process events.

9. The method for determining the degree of fermentation of herbs based on image recognition according to claim 7, characterized in that, The process of collecting chemical reference sample data and performing multi-source calibration between the chemical reference sample data and the online fermentation degree estimation data includes: The state increment variance is obtained by combining the covariance function of the Gaussian dynamic model at the current time with the covariance function of the Gaussian dynamic model at the previous time and the correlation between them. The prediction covariance at the current time is obtained by adding the prediction covariance at the previous time and the current state increment variance. The nonlinear mapping function is linearized once at the predicted mean to obtain the Jacobian matrix; According to the Bayesian online inference rule, the Kalman gain is calculated using the prediction covariance, which includes the state increment variance. The Kalman gain is then used to add the product of the prediction mean and the current observation minus the value of the nonlinear mapping function at the prediction mean to obtain the visual posterior mean of the fermentation degree hidden state. At the same time, the product of the Kalman gain and the Jacobian matrix is ​​subtracted from the product of the identity matrix and the prediction covariance to obtain the visual posterior covariance of the fermentation degree hidden state. Chemical reference sample data is obtained by multiplying the product of the proportional coefficient vector of the latent state of fermentation degree and the product of the latent state of fermentation degree, the chemical reference index bias vector, and the chemical measurement noise. For each chemical reference sample sampling time, the visual posterior mean and visual posterior covariance of the latent state of fermentation degree are used as prior information to perform chemical calibration update in combination with the chemical reference sample data. The chemical calibration Kalman gain is calculated, and the chemical calibration Kalman gain is used to calculate the fermentation degree estimate mean and fermentation degree estimate covariance. The fermentation degree estimate mean and fermentation degree estimate covariance constitute the calibrated fermentation degree estimate data.

10. The method for determining the degree of fermentation of herbs based on image recognition according to claim 1, characterized in that, The process of generating a fermentation degree curve in real time based on the calibrated fermentation degree estimation data and estimating the remaining time to reach the target fermentation degree includes: At each moment during the herbal fermentation cycle, a fermentation degree curve is constructed based on the mean fermentation degree estimate in the calibrated fermentation degree estimate data; The confidence band of the fermentation degree curve is obtained by multiplying the mean of the fermentation degree estimate at each time step by the square root of the normal distribution quantile at the significance level and the covariance of the fermentation degree estimate, and the confidence interval of the fermentation degree at time t is obtained. Set a target fermentation degree and calculate the target remaining difference at the current time. When the target remaining difference is less than or equal to zero, it is determined that the target fermentation degree has been reached or exceeded, and the remaining time to reach the target fermentation degree is recorded as zero. Based on the preset time interval and the estimated average fermentation degree of two adjacent time points, calculate the rate of change of fermentation degree at the current time and the uncertainty of the rate of change of fermentation degree. Under the condition that the target residual difference is greater than zero and the rate of change of fermentation degree is greater than zero, the remaining time estimate is calculated. The uncertainty of the remaining time estimate is obtained by using the uncertainty of the covariance of fermentation degree estimate and the rate of change of fermentation degree at the current time. The confidence interval of the remaining time estimate is calculated based on the uncertainty of the remaining time estimate at the significance level. Align the fermentation degree curve with the confidence band of the fermentation degree curve on the time axis and output the remaining time estimate and its confidence interval, thus completing the real-time generation of the fermentation degree curve and the real-time estimation of the remaining time to reach the target fermentation degree.

Images