Real-time monitoring method and system for milk powder stirring processing

Abstract

The invention provides a real-time monitoring method and system for milk powder stirring processing, and the method comprises the steps: collecting a current frame image of a milk powder material in a stirring container, and obtaining a reference frame image of the current frame image before a preset time interval; calculating a space texture feature set, a time sequence color texture feature set and a dynamic flow field feature set of the current frame image; cascading the space texture feature set, the time sequence color texture feature set and the dynamic flow field feature set to obtain a high-dimensional state vector; calculating a mahalanobis distance between the high-dimensional state vector and a target uniform state cluster core in a pre-constructed state space; when the mahalanobis distance is smaller than a first threshold value, it is judged that the current stirring state is uniform mixing; when the mahalanobis distance is greater than a second threshold value, determining an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, it is determined that mixing is being performed.

Description

Technical Field

[0001] The present application relates to the field of imaging, and in particular to a real-time monitoring method and system for milk powder mixing processing. Background Art

[0002] In the production and processing of milk powder and related powdered products, mixing is a core process for ensuring consistent final product quality. It aims to thoroughly and evenly blend the base milk powder with various nutritional fortifiers to ensure that each serving contains the correct nutritional profile and exhibits good solubility. During production, uneven mixing can easily occur due to differences in material physical properties such as particle size, density, and adhesion, as well as improper mixing equipment and process parameters such as speed and time. This can manifest as material agglomeration, stratification, or the formation of dead zones within the mixing vessel. Relying on operator experience or setting fixed mixing times makes it difficult to ensure consistent product quality across batches. While accurate, offline sampling and analysis methods suffer from significant lag, making them incapable of providing guidance for real-time process control. Machine vision technology, as a non-contact measurement method, is not only more hygienic but also enables real-time monitoring. However, existing vision methods often rely on single or simple image features, such as color histograms and average grayscale values. These features are sensitive to lighting variations and cannot fully capture the complex texture and dynamic changes of powders during mixing. Moreover, the temporal motion information of the material particle flow field is often ignored, resulting in insufficient accuracy in judging the mixing uniformity and identifying abnormal conditions. Summary of the Invention

[0003] The present invention aims to provide a real-time monitoring method and system for milk powder mixing processing, which is used to identify inaccuracies in milk powder mixing using machine vision, comprising the following steps: S1. Acquire a current frame image of the milk powder material in the stirring container, and obtain a reference frame image of the current frame image before a preset time interval; S2. For the L* channel of the current frame image, construct a spatial variation gray-level co-occurrence matrix based on the local gradient information of each pixel, and extract contrast, entropy and energy from it as a spatial texture feature set; S3. Calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set; S4, calculating the dense optical flow field between the current frame image and the reference frame image, extracting the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, to form a dynamic flow field feature set; S5. Concatenate the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in the pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that it is mixing.

[0004] Optionally, constructing a spatial variation gray-level co-occurrence matrix for the L* channel of the current frame image based on local gradient information of each pixel, and extracting contrast, entropy and energy therefrom as a spatial texture feature set, includes: S2.1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

[0005] Optionally, the step of calculating the difference images of the current frame image and the reference frame image in each channel of the CIELab space, extracting the gray-level co-occurrence matrix features of each difference image, and performing weighted summation on the gray-level co-occurrence matrix features using the information entropy of each channel of the current frame as a weight to obtain a temporal color texture feature set includes: S3.1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

[0006] Optionally, the calculation of the dense optical flow field between the current frame image and the reference frame image, the extraction of the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, together forming a dynamic flow field feature set, include: S4.1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.3. Calculate the curl field of the velocity vector field using a difference method; S4.4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

[0007] Optionally, construct the state space by following the steps below: S5.1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state; S5.2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector; S5.3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection; S5.4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

[0008] Optionally, the Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including: S6.1. Obtain the high-dimensional state vector of the current frame ; S6.2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

[0009] The present invention also provides a real-time monitoring system for milk powder mixing processing, comprising the following modules: an acquisition module, configured to acquire a current frame image of the milk powder material in the stirring container and obtain a reference frame image of the current frame image before a preset time interval; A spatial texture feature calculation module is used to construct a spatial variation grayscale co-occurrence matrix based on the local gradient information of each pixel in the L* channel of the current frame image, and extract contrast, entropy and energy from it as a spatial texture feature set; A color and texture feature calculation module is used to calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color and texture feature set; A flow field feature calculation module is used to calculate the dense optical flow field between the current frame image and the reference frame image, extract the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, which together constitute a dynamic flow field feature set; The state judgment module is used to cascade the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in a pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that mixing is in progress.

[0010] Optionally, constructing a spatial variation gray-level co-occurrence matrix for the L* channel of the current frame image based on local gradient information of each pixel, and extracting contrast, entropy and energy therefrom as a spatial texture feature set, includes: S2.1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

[0011] Optionally, the step of calculating the difference images of the current frame image and the reference frame image in each channel of the CIELab space, extracting the gray-level co-occurrence matrix features of each difference image, and performing weighted summation on the gray-level co-occurrence matrix features using the information entropy of each channel of the current frame as a weight to obtain a temporal color texture feature set includes: S3.1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

[0012] Optionally, the calculation of the dense optical flow field between the current frame image and the reference frame image, the extraction of the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, together forming a dynamic flow field feature set, include: S4.1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.3. Calculate the curl field of the velocity vector field using a difference method; S4.4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

[0013] Optionally, construct the state space by following the steps below: S5.1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state; S5.2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector; S5.3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection; S5.4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

[0014] Optionally, the Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including: S6.1. Obtain the high-dimensional state vector of the current frame ; S6.2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

[0015] Compared with existing technologies, this method constructs a spatially variable grayscale co-occurrence matrix based on local gradient information and combines it with temporal color texture features weighted by information entropy. This allows for more accurate assessment of subtle texture differences and color fusion processes caused by changes in particle distribution during the mixing process of milk powder materials. Dynamic flow field analysis based on dense optical flow quantifies the divergence and rotation of the flow field to identify the motion patterns of powder particles, effectively identifying abnormal conditions caused by poor fluidity, such as mixing dead zones and agglomeration. This overcomes the drawback of existing technologies that rely on a single feature and fail to fully reflect mixing quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the first embodiment; Figure 2 A schematic diagram of the overall architecture; Figure 3 Schematic diagram for computing spatial texture feature sets; Figure 4 Schematic diagram for GLCM feature calculation; Figure 5 Schematic diagram for the calculation of dynamic flow field feature set; Figure 6 Schematic diagram for state discrimination. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0018] In the first embodiment, a real-time monitoring method for milk powder stirring processing is provided. Figure 1As shown, the following steps are included: S1. Acquire a current frame image of the milk powder material in the stirring container, and obtain a reference frame image of the current frame image before a preset time interval; The overall process of the present invention is as follows Figure 2 As shown, the first step is to obtain the current frame image and the reference frame image. Specifically, an industrial camera installed above the stirring container continuously captures a video stream with a resolution of 1920×1080 pixels at a rate of, for example, 25 frames per second. The latest frame image in the video stream is used as the current frame image It. The preset time interval is set to a fixed number of frames, for example, 50 frames, and the 50th frame image It minus 50 before the current frame It is defined as the reference frame image Ir. In another embodiment, the reference frame is a fixed frame, for example, one reference frame image for each stage, and the current frame image and the reference frame image are in the same stage.

[0019] S2. For the L* channel of the current frame image, construct a spatial variation gray-level co-occurrence matrix based on the local gradient information of each pixel, and extract contrast, entropy and energy from it as a spatial texture feature set; The current frame image is converted from RGB color space to CIELab color space, and the L* channel component image is extracted. The Sobel operator is used to calculate the gradient amplitude of each pixel on the L* channel image to obtain a gradient map. The grayscale of the L* channel image is quantized into 64 levels. A 64 by 64 grayscale co-occurrence matrix is ​​constructed. When counting pixel pairs, the co-occurrence contribution value of each pixel pair is weighted by its gradient amplitude, that is, the pixel pairs with large gradient amplitude have a greater contribution weight in the matrix. Based on the weighted spatial variation grayscale co-occurrence matrix, its contrast, entropy, and energy are calculated according to the standard formula to form a spatial texture feature set.

[0020] S3. Calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set; The current frame image and the reference frame image are converted to the CIELab color space. Inter-frame difference operations are performed on the three channels L*, a*, and b* to obtain three difference images. For each difference image, its grayscale co-occurrence matrix is ​​calculated, and the four Haralik texture features of contrast, correlation, energy, and homogeneity are extracted from it to obtain three sets of independent texture feature vectors. At the same time, the image information entropy of the three channels L*, a*, and b* of the current frame image is calculated, and recorded as WL, Wa, and Wb respectively. Using the information entropy as the weight, the three sets of texture feature vectors are weighted fused. For example, the final fused contrast feature is equal to the contrast of the L* channel difference image multiplied by WL, plus the contrast of the a* channel difference image multiplied by Wa, plus the contrast of the b* channel difference image multiplied by Wb. The same operation is performed on the other three features, and finally a temporal color texture feature set consisting of four weighted fused eigenvalues ​​is obtained.

[0021] S4, calculating the dense optical flow field between the current frame image and the reference frame image, extracting the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, to form a dynamic flow field feature set; The grayscale images of the current frame and the reference frame are used as input, and the Farneback dense optical flow algorithm is applied to calculate the motion vector of each pixel between the two frames to form a two-dimensional optical flow vector field. The divergence value of each point in the optical flow vector field is calculated using numerical partial derivatives, such as the central difference method, and the mean and variance of all divergence values ​​are further calculated. Similarly, the curl value of each point in the optical flow field is calculated to obtain a curl map. A two-dimensional fast Fourier transform is performed on the curl map, and its energy spectrum is calculated. After normalization, the energy spectrum is regarded as a probability distribution, and its information entropy, that is, the entropy of the curl energy spectrum, is calculated. The three scalar values ​​of the divergence mean, the divergence variance, and the entropy of the curl energy spectrum together constitute the dynamic flow field feature set.

[0022] S5. Concatenate the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in the pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that it is mixing.

[0023] The spatial texture feature set obtained in step S2, the temporal color texture feature set obtained in step S3, and the dynamic flow field feature set obtained in step S4 are concatenated end-to-end to form a unified high-dimensional feature vector. Before implementing the method, an offline training phase is required: a large number of video samples of milk powder materials confirmed to be in a uniformly mixed state are collected, and the above-mentioned high-dimensional feature vectors are extracted for each sample to form a set of feature vectors for the target uniform state. The mean vector and covariance matrix of this set are calculated, where the mean vector is the core of the target uniform state cluster. During real-time monitoring, the Mahalanobis distance between the newly calculated high-dimensional state vector and the pre-stored target uniform state cluster core is calculated. A first threshold T1 and a second threshold T2 are pre-set, for example, T1 is set to 1.5 and T2 is set to 3.0. If the calculated Mahalanobis distance is less than 1.5, the mixture is determined to be uniform; if the Mahalanobis distance is greater than 3.0, it is determined to be in an abnormal state such as agglomeration or stratification; if the Mahalanobis distance is between 1.5 and 3.0, it is determined to be mixing.

[0024] In order to calculate the spatial inhomogeneity formed by milk powder particles or lumps in the liquid, in a preferred embodiment, the spatial variation gray-level co-occurrence matrix is ​​constructed based on the local gradient information of each pixel in the L* channel of the current frame image, and contrast, entropy and energy are extracted from it as a spatial texture feature set, including: S2.1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

[0025] The L* channel of CIELab space is perceived by the human eye as brightness. It is less sensitive to changes in the brightness of milk powder particles than to changes in color, making it more suitable for texture analysis. Calculating gradients using the Sobel operator can identify areas of image brightness abrupt changes caused by the edges of milk powder particles or clumps. Quantizing the continuous gradient direction to a limited number, such as eight, simplifies calculations while preserving key texture direction information. Constructing pixel pairs along these directions and counting their brightness values ​​creates a spatially varying grayscale co-occurrence matrix that captures texture correlations along specific directions. This is very effective for identifying undissolved milk powder strips or lumps.

[0026] Taking a 1024×768 pixel L* channel image as an example, the gradient direction of each pixel is first calculated and quantized to eight discrete directions, such as 0 degrees, 45 degrees, 90 degrees, and 135 degrees. For the pixel points with coordinates 300 and 400 in the image, if the quantized gradient direction is 45 degrees, then find the neighboring pixels with a preset distance of 5 pixels, such as 5 pixels, and a direction of 45 degrees, that is, pixels with coordinates approximately 303 and 403, to form a pixel pair. Assuming that the L* values ​​of these two pixels are 85 and 90 respectively, the position counts corresponding to 85 and 90 in the spatial variation grayscale co-occurrence matrix are increased by one. After traversing all pixels, a 256×256 matrix representing the global spatial texture is obtained. After normalizing this matrix, the calculated contrast may be 1.2, the entropy is 7.5, and the energy is 0.02. These three values ​​together constitute the spatial texture feature set, such as Figure 3 As shown, it can reflect the degree of agglomeration and the roughness of the distribution of milk powder in the current frame.

[0027] In a preferred embodiment, the calculation of the difference image of each channel of the current frame image and the reference frame image in the CIELab space, the extraction of the gray level co-occurrence matrix features of each difference image, and the weighted summation of the gray level co-occurrence matrix features using the information entropy of each channel of the current frame as a weight to obtain a temporal color texture feature set includes: S3.1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

[0028] The temporal color and texture feature set is mainly used to discover the dynamic changes of color and texture during the mixing process. In CIELab space, the difference in the L* channel represents the change in brightness, the difference in the a* channel represents the change in red and green colors, and the difference in the b* channel represents the change in yellow and blue colors, which represents the formation process of milky white liquid. The gray level co-occurrence matrix is ​​calculated for these difference images, such as Figure 4As shown in Figure 2, its contrast feature measures the clarity of the changing region, while the homogeneity feature measures the uniformity of the changing region.

[0029] Specifically, suppose the reference frame is clear water, and the current frame is the moment milk powder is poured in. In the b* channel, the yellow-blue channel, the difference image will have large non-zero regions due to the pale yellow color of the milk powder. The gray-level co-occurrence matrix calculated for this difference image may yield a high contrast value, such as 2.5, and a low homogeneity value, such as 0.4, indicating a dramatic and uneven color change. Simultaneously, the information entropy of the three channels, L*, a*, and b*, for the current frame may be calculated, yielding values ​​of 7.2, 5.1, and 6.5, respectively. The total entropy is 18.8. Weighted contrast is the sum of the contrasts of the three difference images, L*, a*, and b*, weighted by 7.2 to 18.8, 5.1 to 18.8, and 6.5 to 18.8. Channels with greater information content contribute more to the final feature, allowing the feature to reflect the overall visual changes in the mixed state.

[0030] In a preferred embodiment, the dense optical flow field between the current frame image and the reference frame image is calculated, and the divergence mean and variance of the optical flow vector field and the entropy of the curl energy spectrum are extracted to form a dynamic flow field feature set, including: S4.1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.3. Calculate the curl field of the velocity vector field using a difference method; S4.4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

[0031] like Figure 5As shown, the dynamic flow feature set includes at least the mean and variance of the divergence field and the entropy of the curl energy spectrum. The dense optical flow method estimates the motion of each pixel in the image, thereby approximating the two-dimensional velocity field of the liquid and suspended particles. Divergence describes the convergence and divergence of the flow field. A positive divergence mean may indicate that milk powder clumps are dissolving and dispersing, while the variance indicates the uniformity of this dispersion process across the entire field of view. Curl describes the local rotational intensity of the flow field and is directly related to the turbulence and vortices generated by stirring. Specifically, taking the current and previous frames as input, the dense optical flow algorithm outputs a velocity vector field of the same size as the image. For example, near a stirring rod, the velocity vector of a pixel may be larger and point in the direction of rotation. By numerically differentiating the velocity field, the divergence and curl fields are obtained. Assuming the calculated divergence field has a mean of 0.03 and a variance of 0.1, this indicates that the liquid is slightly expanding overall, with the degree of expansion varying from location to location. A two-dimensional Fourier transform of the curl field yields its energy spectrum, which represents the distribution of vortex energy at different scales. If the stirring is vigorous, the energy is widely distributed across high, medium, and low frequencies, and the calculated entropy of the energy spectrum can be as high as 6.9. However, if the stirring is slow, forming a large, stable vortex, the energy is concentrated at low frequencies, and the entropy can be as low as 2.5.

[0032] In a preferred embodiment, constructing the state space is achieved by the following steps: S5.1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state; S5.2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector; S5.3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection; S5.4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

[0033] To calibrate the system, a standard target state is defined by learning from a large number of samples known to be well-mixed. By collecting multiple independent mixing videos, the model's generalization ability is ensured, avoiding overfitting to specific experimental conditions. Features are extracted only from the final homogenization phase of each video to ensure that the learned target state is stable and ideal. The extracted high-dimensional state vector incorporates all the aforementioned features, including spatial texture, temporal color texture, and dynamic flow field, forming a signature of the mixing state. For example, 20 mixing videos were collected. From the last 20 seconds of each video, where homogenization was confirmed, 5 frames were extracted per second, for a total of 100 frames, resulting in 2000 sample frames. For each frame, a state vector containing seven eigenvalues ​​is calculated, such as spatial contrast, temporal homogeneity, and mean divergence. This results in a dataset of 2000 rows and 7 columns. Based on this dataset, a 7-dimensional mean vector μ is calculated, which represents the most typical characteristic representation of a well-mixed state. At the same time, a 7x7 covariance matrix Σ is calculated, which describes the normal fluctuation range of each eigenvalue in the uniform state and the correlation between them. This mean vector and covariance matrix together define the target uniform region in the high-dimensional feature space.

[0034] In a preferred embodiment, the Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including: S6.1. Obtain the high-dimensional state vector of the current frame ; S6.2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

[0035] Calculate the high-dimensional state vector of each frame collected in real time. This vector represents the current mixed state. Calculate the Mahalanobis distance between this vector and the predefined target uniform state center μ. A small Mahalanobis distance means that the current state is statistically very close to the ideal uniform state. Assume that the state vector calculated for the current frame is is [1.1,7.2,0.5,0.6,2.1,0.9,4.8]. Using the pre-calculated mean vector μ and the inverse of the covariance matrix , the value of the Mahalanobis distance D calculated by the formula is 9.7. Two thresholds are set in advance, for example, the first threshold is 4.0 and the second threshold is 8.0. These two thresholds are obtained by analyzing the Mahalanobis distance distribution of a large number of uniform samples and non-uniform samples, and selecting the boundaries that can best distinguish them. Since the calculated distance of 9.7 is greater than the second threshold of 8.0, the current state is judged to be unevenly mixed. If the calculated distance is 3.1, it is less than the first threshold of 4.0, and the state will be judged to be uniformly mixed. If the distance is 6.5, which is between the two thresholds, the state is judged to be mixed, such as Figure 6 shown.

[0036] In a second embodiment, a real-time monitoring system for milk powder mixing processing is provided, comprising the following modules: an acquisition module, configured to acquire a current frame image of the milk powder material in the stirring container and obtain a reference frame image of the current frame image before a preset time interval; A spatial texture feature calculation module is used to construct a spatial variation grayscale co-occurrence matrix based on the local gradient information of each pixel in the L* channel of the current frame image, and extract contrast, entropy and energy from it as a spatial texture feature set; A color and texture feature calculation module is used to calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color and texture feature set; A flow field feature calculation module is used to calculate the dense optical flow field between the current frame image and the reference frame image, extract the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, which together constitute a dynamic flow field feature set; The state judgment module is used to cascade the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in a pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that mixing is in progress.

[0037] In a preferred embodiment, for the L* channel of the current frame image, constructing a spatial variation gray level co-occurrence matrix based on the local gradient information of each pixel, and extracting contrast, entropy and energy as a spatial texture feature set, includes: S2.1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

[0038] In a preferred embodiment, the calculation of the difference image of each channel of the current frame image and the reference frame image in the CIELab space, the extraction of the gray level co-occurrence matrix features of each difference image, and the weighted summation of the gray level co-occurrence matrix features using the information entropy of each channel of the current frame as a weight to obtain a temporal color texture feature set includes: S3.1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

[0039] In a preferred embodiment, the dense optical flow field between the current frame image and the reference frame image is calculated, and the divergence mean and variance of the optical flow vector field and the entropy of the curl energy spectrum are extracted to form a dynamic flow field feature set, including: S4.1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.3. Calculate the curl field of the velocity vector field using a difference method; S4.4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

[0040] In a preferred embodiment, constructing the state space is achieved by the following steps: S5.1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state; S5.2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector; S5.3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection; S5.4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

[0041] Optionally, the Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including: S6.1. Obtain the high-dimensional state vector of the current frame ; S6.2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

[0042] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0043] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0044] The above describes in detail the method and electronic device for providing commodity object information provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be construed as limiting this application.

Claims

1. A real-time monitoring method for milk powder stirring processing, characterized in that: The following steps are involved: S1. Acquire a current frame image of the milk powder material in the stirring container, and obtain a reference frame image of the current frame image before a preset time interval; S2. For the L* channel of the current frame image, construct a spatial variation gray-level co-occurrence matrix based on the local gradient information of each pixel, and extract contrast, entropy and energy from it as a spatial texture feature set; S3. Calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set; S4, calculating the dense optical flow field between the current frame image and the reference frame image, extracting the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, to form a dynamic flow field feature set; S5. Concatenate the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in the pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that it is mixing.

2. The method according to claim 1, characterized in that The L* channel of the current frame image is constructed based on the local gradient information of each pixel in the spatial variation gray level co-occurrence matrix, and contrast, entropy and energy are extracted as a spatial texture feature set, including: S2.

1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.

2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.

3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.

4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.

5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

3. The method according to claim 1, characterized in that The method comprises calculating the difference images of each channel of the current frame image and the reference frame image in the CIELab space, extracting the gray-level co-occurrence matrix features of each difference image, and using the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set, including: S3.

1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.

2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.

3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.

4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.

5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

4. The method according to claim 1, wherein The dense optical flow field between the current frame image and the reference frame image is calculated, and the divergence mean and variance of the optical flow vector field and the entropy of the curl energy spectrum are extracted to form a dynamic flow field feature set, including: S4.

1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.

2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.

3. Calculate the curl field of the velocity vector field using a difference method; S4.

4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.

5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

5. The method according to claim 1, wherein Constructing the state space is achieved through the following steps: S5.

1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state; S5.

2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector; S5.

3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection; S5.

4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

6. The method according to claim 5, characterized in that The Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including: S6.

1. Obtain the high-dimensional state vector of the current frame ; S6.

2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

7. A real-time monitoring system for milk powder mixing processing, characterized in that: Includes the following modules: an acquisition module, configured to acquire a current frame image of the milk powder material in the stirring container and obtain a reference frame image of the current frame image before a preset time interval; A spatial texture feature calculation module is used to construct a spatial variation grayscale co-occurrence matrix based on the local gradient information of each pixel in the L* channel of the current frame image, and extract contrast, entropy and energy from it as a spatial texture feature set; A color and texture feature calculation module is used to calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color and texture feature set; A flow field feature calculation module is used to calculate the dense optical flow field between the current frame image and the reference frame image, extract the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, which together constitute a dynamic flow field feature set; The state judgment module is used to cascade the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in a pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that mixing is in progress.

8. The system according to claim 7, characterized in that The L* channel of the current frame image is constructed based on the local gradient information of each pixel in the spatial variation gray level co-occurrence matrix, and contrast, entropy and energy are extracted as a spatial texture feature set, including: S2.

1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.

2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.

3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.

4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.

5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

9. The system according to claim 7, wherein: The method comprises calculating the difference images of each channel of the current frame image and the reference frame image in the CIELab space, extracting the gray-level co-occurrence matrix features of each difference image, and using the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set, including: S3.

1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.

2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.

3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.

4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.

5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

10. The system according to claim 7, wherein: The dense optical flow field between the current frame image and the reference frame image is calculated, and the divergence mean and variance of the optical flow vector field and the entropy of the curl energy spectrum are extracted to form a dynamic flow field feature set, including: S4.

1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.

2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.

3. Calculate the curl field of the velocity vector field using a difference method; S4.

4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.

5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

Images