A real-time monitoring method and system for milk powder stirring processing

Abstract

The application provides a real-time monitoring method and system for milk powder stirring processing, acquires a current frame image of milk powder material in a stirring container, and acquires a reference frame image before a preset time interval of the current frame image; calculates a spatial texture feature set, a time sequence color texture feature set and a dynamic flow field feature set of the current frame image; cascades the spatial texture feature set, the time sequence color texture feature set and the dynamic flow field feature set to obtain a high-dimensional state vector; calculates Mahalanobis distance of the high-dimensional state vector in a state space constructed in advance and a target uniform state cluster core; when the Mahalanobis distance is less than a first threshold value, it is determined that the current stirring state is mixed uniformly; when the Mahalanobis distance is greater than a second threshold value, it is determined that the state is abnormal; and when the Mahalanobis distance is between the first threshold value and the second threshold value, it is determined that the mixing is in progress.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of images, and particularly relates to a real-time monitoring method and system for milk powder stirring processing. BACKGROUND

[0002] In the production and processing of milk powder and related powder products, stirring and mixing is a core process to ensure the uniformity of the quality of the final product, aiming to fully and uniformly mix the base milk powder with various nutritional fortifiers to ensure that each product contains accurate nutritional ingredients and has good brewing solubility. In production, due to the differences in physical properties of the materials such as particle size, density, and adhesion, and improper settings of stirring equipment and process parameters such as rotation speed and time, it is easy to cause uneven mixing, which is specifically manifested in the agglomeration of material clumps, the stratification of ingredients, or the formation of dead zones with poor flowability in the stirring container. Relying on the experience observation of the operator or setting a fixed stirring time, it is difficult to ensure the stability of the product quality between different batches. Although the offline sampling analysis method is accurate, it has serious lag and cannot provide guidance for real-time process control. As a non-contact measurement method, machine vision technology is not only more sanitary but also can be monitored in real time, but existing visual methods mostly rely on single or simple image features such as color histogram and average gray level, which are sensitive to light changes and difficult to fully depict the complex texture and dynamic changes of the powder in the mixing process. Moreover, the time sequence motion information of the material particle flow field is often ignored, resulting in insufficient accuracy in the discrimination of mixing uniformity and the identification of abnormal states. SUMMARY

[0003] The purpose of the present application is to provide a real-time monitoring method and system for milk powder stirring processing, which is used to identify the problem of inaccuracy in milk powder stirring using machine vision, comprising the following steps:

[0004] S1, acquiring a current frame image of the milk powder material in the stirring container, and obtaining a reference frame image before a preset time interval of the current frame image;

[0005] S2, constructing a spatial variant gray level co-occurrence matrix based on the local gradient information of each pixel for the L* channel of the current frame image, and extracting contrast, entropy, and energy as a spatial texture feature set from the spatial variant gray level co-occurrence matrix;

[0006] S3, calculating difference images of the current frame image and the reference frame image in each channel of the CIELab space, extracting gray level co-occurrence matrix features of each difference image, and using the information entropy of each channel of the current frame as a weight to weight and sum the gray level co-occurrence matrix features to obtain a time sequence color texture feature set;

[0007] S4. Calculate the dense optical flow field between the current frame image and the reference frame image, extract the divergence mean and variance of the optical flow vector field, and the entropy of the vorticity energy spectrum, to form a dynamic flow field feature set;

[0008] S5. Cascade the spatial texture feature set, the time-series color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the target uniform state cluster core in the pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current mixing state is uniform mixing; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; and when the Mahalanobis distance is between the first threshold and the second threshold, determine that it is being mixed.

[0009] Optionally, the L* channel of the current frame image is based on the local gradient information of each pixel to construct a spatial variant gray level co-occurrence matrix, and the contrast, entropy and energy are extracted therefrom as a spatial texture feature set, including:

[0010] S2.1. Convert the current frame image from RGB space to CIELab space and extract the L* channel image;

[0011] S2.2. Calculate the gradient amplitude and gradient direction of each pixel point in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions;

[0012] S2.3. For each pixel point in the L* channel image, find the neighborhood pixels of a predetermined distance in the quantized gradient direction to form a pixel pair;

[0013] S2.4. Construct a spatial variant gray level co-occurrence matrix according to the L* values of all pixel pairs and perform normalization processing;

[0014] S2.5. Calculate the contrast, entropy and energy based on the normalized spatial variant gray level co-occurrence matrix as the spatial texture feature set.

[0015] Optionally, the difference images of the current frame image and the reference frame image in each channel of the CIELab space are calculated, the gray level co-occurrence matrix features of each difference image are extracted, and the information entropy of each channel of the current frame is used as a weight to weight and sum the gray level co-occurrence matrix features to obtain a time-series color texture feature set, including:

[0016] S3.1. Convert the current frame image and the reference frame image to CIELab space to obtain L*, a* and b* channels;

[0017] S3.2. For each L*, a* and b* channel, calculate the difference image of the current frame image and the reference frame image, respectively;

[0018] S3.3, respectively calculating a gray level co-occurrence matrix for the three difference images, and extracting contrast and homogeneity features from each matrix;

[0019] S3.4, calculating information entropy of each of the three channels of the current frame image L*, a* and b*;

[0020] S3.5, using the information entropy of each channel as a weight, performing cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the time-series color texture feature set.

[0021] Optionally, the calculation of the dense optical flow field between the current frame image and the reference frame image, the extraction of the mean and variance of the divergence of the optical flow vector field, and the entropy of the energy spectrum of the vorticity, together constitute the dynamic flow field feature set, comprising:

[0022] S4.1, calculating the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field of each pixel point;

[0023] S4.2, calculating the divergence field of the velocity vector field using the difference method, and calculating the mean and variance of the divergence field;

[0024] S4.3, calculating the vorticity field of the velocity vector field using the difference method;

[0025] S4.4, performing two-dimensional Fourier transform on the vorticity field and calculating the energy spectrum;

[0026] S4.5, normalizing the energy spectrum into a probability distribution and calculating the information entropy as the entropy of the vorticity energy spectrum.

[0027] Optionally, a state space is constructed by the following steps:

[0028] S5.1, collecting multiple independent milk powder stirring process videos that eventually reach a mixed uniform state;

[0029] S5.2, extracting at least one frame of image from the mixed uniform stage of each video, and extracting a high-dimensional state vector;

[0030] S5.3, collecting all the extracted high-dimensional state vectors together, and calculating the mean vector μ and the covariance matrix Σ of the set;

[0031] S5.4, the target uniform state cluster core is the mean vector μ, and the Mahalanobis distance calculation used in state discrimination is based on the inverse matrix of the covariance matrix Σ.

[0032] Optionally, the Mahalanobis distance calculation used in state discrimination is based on the inverse matrix of the covariance matrix Σ, specifically comprising:

[0033] S6.1. Obtain the high-dimensional state vector of the current frame ;

[0034] S6.2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

[0035] The present invention also provides a real-time monitoring system for milk powder mixing processing, comprising the following modules:

[0036] an acquisition module, configured to acquire a current frame image of the milk powder material in the stirring container and obtain a reference frame image of the current frame image before a preset time interval;

[0037] A spatial texture feature calculation module is used to construct a spatial variation grayscale co-occurrence matrix based on the local gradient information of each pixel in the L* channel of the current frame image, and extract contrast, entropy and energy from it as a spatial texture feature set;

[0038] A color and texture feature calculation module is used to calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color and texture feature set;

[0039] A flow field feature calculation module is used to calculate the dense optical flow field between the current frame image and the reference frame image, extract the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, which together constitute a dynamic flow field feature set;

[0040] The state judgment module is used to cascade the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in a pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that mixing is in progress.

[0041] Optionally, constructing a spatial variation gray-level co-occurrence matrix for the L* channel of the current frame image based on local gradient information of each pixel, and extracting contrast, entropy and energy therefrom as a spatial texture feature set, includes:

[0042] S2.1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image;

[0043] S2.2, calculate the gradient amplitude and gradient direction of each pixel point in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions;

[0044] S2.3, for each pixel point in the L* channel image, find the neighborhood pixels of a predetermined distance in the quantized gradient direction to form a pixel pair;

[0045] S2.4, construct a spatial variant gray level co-occurrence matrix according to the L* values of all pixel pairs, and perform normalization processing;

[0046] S2.5, based on the normalized spatial variant gray level co-occurrence matrix, calculate the contrast, entropy and energy as the spatial texture feature set.

[0047] Optionally, the difference images of the current frame image and the reference frame image in each channel of the CIELab space are calculated, the gray level co-occurrence matrix features of each difference image are extracted, and the information entropy of each channel of the current frame is used as a weight to weight and sum the gray level co-occurrence matrix features, to obtain a time sequence color texture feature set, including:

[0048] S3.1, convert the current frame image and the reference frame image into CIELab space to obtain L*, a* and b* three channels;

[0049] S3.2, for each channel of L*, a* and b*, calculate the difference images of the current frame image and the reference frame image respectively;

[0050] S3.3, calculate the gray level co-occurrence matrix for the three difference images respectively, and extract the contrast and homogeneity features from each matrix;

[0051] S3.4, calculate the information entropy of each of the L*, a* and b* channels of the current frame image;

[0052] S3.5, use the information entropy of each channel as a weight to cross-channel weight and sum the contrast and homogeneity features extracted in S3.3, to obtain the weighted contrast and homogeneity features, which together constitute the time sequence color texture feature set.

[0053] Optionally, the dense optical flow field between the current frame image and the reference frame image is calculated, the divergence mean and variance of the optical flow vector field, and the entropy of the vorticity energy spectrum are extracted, to constitute a dynamic flow field feature set, including:

[0054] S4.1, calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field of each pixel point;

[0055] S4.2, calculate the divergence field of the velocity vector field by using the difference method, and calculate the mean and variance of the divergence field;

[0056] S4.3, calculate the curl field of the velocity vector field by using the difference method;

[0057] S4.4, perform two-dimensional Fourier transform on the curl field, and calculate the energy spectrum;

[0058] S4.5, normalize the energy spectrum into a probability distribution, and calculate the information entropy as the entropy of the curl energy spectrum.

[0059] Optionally, a state space is constructed, which is realized by the following steps:

[0060] S5.1, collect multiple independent milk powder stirring process videos that finally reach a mixed uniform state;

[0061] S5.2, extract at least one frame of image from the mixed uniform stage of each video, and extract a high-dimensional state vector;

[0062] S5.3, collect all the extracted high-dimensional state vectors together, and calculate the mean vector μ and the covariance matrix Σ of the set;

[0063] S5.4, the target uniform state cluster core is the mean vector μ, and the Mahalanobis distance calculation used in the state discrimination is based on the inverse matrix of the covariance matrix Σ.

[0064] Optionally, the Mahalanobis distance calculation used in the state discrimination is based on the inverse matrix of the covariance matrix Σ, and specifically includes:

[0065] S6.1, obtain the high-dimensional state vector of the current frame ;

[0066] S6.2, calculate the Mahalanobis distance by using the pre-calculated mean vector and the inverse of the covariance matrix ; wherein sqrt() represents square root operation.

[0067] Compared with the prior art, the present application can more accurately judge the subtle texture difference caused by the change of particle distribution and the process of color fusion of the milk powder material in the mixing process by constructing a spatial variation gray level co-occurrence matrix based on local gradient information and combining a time sequence color texture feature with an information entropy weight. Based on the dynamic flow field analysis of dense optical flow, the motion mode of powder particles is identified by quantifying the divergence and curl information of the flow field, so that the abnormal states such as stirring dead zone and agglomeration caused by poor fluidity can be effectively identified. The defects of the prior art that rely on a single feature and cannot comprehensively reflect the mixing quality are overcome. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 a flowchart for the first embodiment;

[0069] Figure 2 a schematic diagram of the overall architecture;

[0070] Figure 3 a schematic diagram for spatial texture feature set calculation;

[0071] Figure 4 a schematic diagram for GLCM feature calculation;

[0072] Figure 5 a schematic diagram for dynamic flow field feature set calculation;

[0073] Figure 6 a schematic diagram for state discrimination. DETAILED DESCRIPTION

[0074] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for the user to choose authorization or refusal.

[0075] In the first embodiment, a real-time monitoring method for milk powder stirring processing is provided, as shown in Figure 1 The method comprises the following steps:

[0076] S1, acquiring a current frame image of milk powder material in a stirring container, and obtaining a reference frame image before a preset time interval of the current frame image;

[0077] The overall process of the present application is as shown in Figure 2As shown, first, the current frame image and the reference frame image are acquired, specifically, by an industrial camera installed above the stirring container, a video stream with a resolution of 1920x1080 pixels is continuously collected at a rate of, for example, 25 frames per second. The latest frame image in the video stream is taken as the current frame image It. A preset time interval is set as a fixed number of frames, for example, 50 frames, and the 50th frame image It-50 before the current frame It is defined as the reference frame image Ir. In yet another embodiment, the reference frame is a fixed frame, for example, one reference frame image per stage, and the current frame image and the reference frame image are in the same stage.

[0078] S2, for the L* channel of the current frame image, based on the local gradient information of each pixel, a spatial variant gray level co-occurrence matrix is constructed, and contrast, entropy and energy are extracted therefrom as a spatial texture feature set;

[0079] The current frame image is converted from the RGB color space to the CIELab color space, and the L* channel component image is extracted. The gradient amplitude of each pixel point on the L* channel image is calculated using the Sobel operator to obtain a gradient map. The gray level of the L* channel image is quantized into 64 levels. A 64x64 gray level co-occurrence matrix is constructed. When counting pixel pairs, the contribution value of each pixel pair is weighted using its gradient amplitude, that is, the pixel pair with larger gradient amplitude has greater contribution weight in the matrix. Based on the weighted spatial variant gray level co-occurrence matrix, the contrast, entropy and energy of the three texture measures are calculated according to the standard formula to form a spatial texture feature set.

[0080] S3, the difference images of the current frame image and the reference frame image in the CIELab space are calculated, the gray level co-occurrence matrix features of each difference image are extracted, and the information entropy of each channel of the current frame is used as a weight to sum the gray level co-occurrence matrix features, to obtain a time sequence color texture feature set;

[0081] The current frame image and the reference frame image are converted to CIELab color space. Inter-frame difference operation is performed on the L*, a*, b* three channels respectively to obtain three difference images. For each difference image, its gray level co-occurrence matrix is calculated, and four Haralick texture features of contrast, correlation, energy and homogeneity are extracted from it, thereby obtaining three independent texture feature vectors. At the same time, the image information entropy of each of the L*, a*, b* three channels of the current frame image is calculated, and is denoted as WL, Wa, Wb respectively. The information entropy is used as a weight to perform weighted fusion on the three groups of texture feature vectors, for example, the final fused contrast feature is equal to the contrast of the L* channel difference image multiplied by WL, plus the contrast of the a* channel difference image multiplied by Wa, plus the contrast of the b* channel difference image multiplied by Wb. The same operation is performed on the other three features, and finally a time-series color texture feature set composed of four weighted and fused feature values is obtained.

[0082] S4, calculate the dense optical flow field between the current frame image and the reference frame, extract the mean value and variance of the divergence of the optical flow vector field, and the entropy of the energy spectrum of the vorticity, to jointly constitute a dynamic flow field feature set;

[0083] The gray scale images of the current frame and the reference frame are taken as inputs, and a Farneback dense optical flow algorithm is applied to calculate the motion vector of each pixel point between the two frames to form a two-dimensional optical flow vector field. The numerical partial derivative, such as the central difference method, is used to calculate the divergence value of each point in the optical flow vector field, and the mean value and variance of all divergence values are further calculated. Similarly, the vorticity value of each point in the optical flow field is calculated to obtain a vorticity map. A two-dimensional fast Fourier transform is performed on the vorticity map, and its energy spectrum is calculated. After normalization processing of the energy spectrum, it is regarded as a probability distribution, and its information entropy, i.e. the entropy of the vorticity energy spectrum, is calculated. The three scalar values of the mean value of the divergence, the variance of the divergence and the entropy of the vorticity energy spectrum jointly constitute a dynamic flow field feature set.

[0084] S5, concatenate the spatial texture feature set, the time-series color texture feature set and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the target uniform state cluster core in the pre-constructed state space; when the Mahalanobis distance is less than a first threshold value, it is determined that the current mixing state is uniform mixing; when the Mahalanobis distance is greater than a second threshold value, it is determined that it is an abnormal state; when the Mahalanobis distance is between the first threshold value and the second threshold value, it is determined that it is being mixed.

[0085] The spatial texture feature set obtained in step S2, the time-series color texture feature set obtained in step S3 and the dynamic flow field feature set obtained in step S4 are concatenated to form a unified high-dimensional feature vector. Before the method is implemented, an offline training phase is required: a large number of video samples of milk powder material in a confirmed homogeneous state are collected, and the high-dimensional feature vector is extracted from each sample to form a feature vector set of the target homogeneous state. The mean vector and the covariance matrix of the set are calculated, wherein the mean vector is the core of the target homogeneous state cluster. In real-time monitoring, the Mahalanobis distance between the newly calculated high-dimensional state vector and the pre-stored core of the target homogeneous state cluster is calculated. The first threshold T1 and the second threshold T2 are pre-set, for example, T1 is set to 1.5 and T2 is set to 3.0. If the calculated Mahalanobis distance is less than 1.5, it is determined that the mixing is homogeneous; if the Mahalanobis distance is greater than 3.0, it is determined that the state is abnormal, such as agglomeration or stratification; and if the Mahalanobis distance is between 1.5 and 3.0, it is determined that the mixing is in progress.

[0086] In order to calculate the spatial inhomogeneity of the milk powder particles or lumps formed in the liquid. In a preferred embodiment, the L* channel of the current frame image is constructed based on the local gradient information of each pixel to construct a spatial variation gray level co-occurrence matrix, and the contrast, entropy and energy are extracted therefrom as a spatial texture feature set, including:

[0087] S2.1, converting the current frame image from RGB space to CIELab space, and extracting the L* channel image;

[0088] S2.2, calculating the gradient amplitude and gradient direction of each pixel point in the L* channel image using the Sobel operator, and quantizing the gradient direction into a plurality of discrete directions;

[0089] S2.3, for each pixel point in the L* channel image, finding a neighborhood of pixels with a preset distance in the quantized gradient direction to form a pixel pair;

[0090] S2.4, constructing a spatial variation gray level co-occurrence matrix according to the L* values of all pixel pairs, and performing normalization processing;

[0091] S2.5, calculating the contrast, entropy and energy based on the normalized spatial variation gray level co-occurrence matrix as the spatial texture feature set.

[0092] The L* channel of CIELab space is perceived by human eyes as luminance, which is insensitive to color change and thus more suitable for texture analysis. Gradient is calculated by Sobel operator, which can identify the area of luminance mutation caused by the edge of milk powder particles or clumps in the image. Quantizing the continuous gradient direction into a limited number of directions, for example, eight directions, simplifies the calculation while preserving the main texture direction information. Along these directions, pixel pairs are constructed and their luminance values are counted, forming a spatially variant gray level co-occurrence matrix which can capture the texture correlation in a specific direction, which is very effective for identifying unsolved strip or block of milk powder.

[0093] Take a 1024x768 pixel L* channel image as an example, first calculate the gradient direction of each pixel point, and quantize it to eight discrete directions such as 0 degrees, 45 degrees, 90 degrees, and 135 degrees. For the pixel points with coordinates 300, 400 in the image, if their quantized gradient direction is 45 degrees, then find the neighborhood pixels with a distance of, for example, 5 pixels and a direction of 45 degrees from them, that is, the pixels with coordinates approximately 303, 403, to form a pixel pair. Assuming that the L* values of the two pixels are 85 and 90 respectively, then the position corresponding to 85, 90 in the spatially variant gray level co-occurrence matrix is counted by one. After traversing all the pixel points, a 256x256 matrix representing the global spatial texture is obtained. After normalizing the matrix, the calculated contrast may be 1.2, the entropy may be 7.5, and the energy may be 0.02, which together form a spatial texture feature set as shown in Figure 3

[0094] In a preferred embodiment, the difference images of the current frame image and the reference frame image in each channel of CIELab space are calculated, the gray level co-occurrence matrix features of each difference image are extracted, and the information entropy of each channel of the current frame is used as a weight to weight and sum the gray level co-occurrence matrix features, to obtain a time-series color texture feature set, including:

[0095] S3.1, converting the current frame image and the reference frame image into CIELab space to obtain three channels of L*, a*, and b*;

[0096] S3.2, for each channel of L*, a*, and b*, calculating the difference images of the current frame image and the reference frame image respectively;

[0097] S3.3, calculating the gray level co-occurrence matrix for each of the three difference images, and extracting the contrast and homogeneity features from each matrix;

[0098] S3.4, calculating the information entropy of each of the L*, a*, and b* channels of the current frame image; ​

[0099] S3.5, using the information entropy of each channel as weight, cross-channel weighted sum of the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the time-sequential color texture feature set.

[0100] The time-sequential color texture feature set is mainly used to find the dynamic changes of color and texture in the mixing process. In the CIELab space, the difference of L* channel represents the change of brightness, the difference of a* channel represents the change of red and green color, and the difference of b* channel represents the change of yellow and blue color, which represents the formation process of the milk-white liquid. The gray level co-occurrence matrix is calculated for these difference images as shown in Figure 4 The contrast feature measures the sharpness of the change area, and the homogeneity feature measures the uniformity of the change area.

[0101] Specifically, assuming that the reference frame is clear water and the current frame is the moment when the milk powder is just poured in. In the b* channel, i.e. the yellow and blue channel, the difference image will have a large non-zero area due to the light yellow color of the milk powder. Therefore, the gray level co-occurrence matrix calculated for the difference image may obtain a higher contrast value, for example, 2.5, and a lower homogeneity value, for example, 0.4, indicating that the color change is intense and uneven. At the same time, the information entropy of each of the L*, a*, and b* channels of the current frame is calculated, and the obtained values may be 7.2, 5.1, and 6.5, respectively. The total entropy is 18.8. The weighted contrast is the weighted sum of the contrast of the L*, a*, and b* difference images according to the weights of 7.2 to 18.8, 5.1 to 18.8, and 6.5 to 18.8, respectively. The channel with more information contributes more to the final feature, so that the feature can reflect the overall visual change of the mixing state.

[0102] In a preferred embodiment, the method further comprises:

[0103] S4.1, calculating the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field of each pixel point;

[0104] S4.2, calculating the divergence field of the velocity vector field using the difference method, and calculating the mean and variance of the divergence field;

[0105] S4.3, calculating the curl field of the velocity vector field using the difference method;

[0106] S4.4, performing two-dimensional Fourier transform on the curl field, and calculating the energy spectrum;

[0107] S4.5, normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the vorticity energy spectrum.

[0108] As shown in Figure 5 , the features of the dynamic flow field feature set include at least the mean and variance of the divergence field, and the entropy of the vorticity energy spectrum. The dense optical flow method can estimate the motion of each pixel in the image, thereby approximating the two-dimensional velocity field of the liquid and suspended particles. The divergence describes the convergence and divergence of the flow field, and a positive mean divergence value may indicate that the milk powder clumps are dissolving and dispersing, while the variance indicates the uniformity of this dispersing process throughout the field of view. The vorticity describes the local rotation intensity of the flow field, which is directly related to the turbulence and vortex generated by stirring. Specifically, the dense optical flow algorithm outputs a velocity vector field with the same size as the image using the current frame and the previous frame as input. For example, near the stirring rod, the velocity vector of a pixel point may be large and point in the direction of rotation. By numerically differentiating the velocity field, the divergence field and the vorticity field are obtained. Assuming that the calculated mean value of the divergence field is 0.03 and the variance is 0.1, it indicates that the liquid is slightly expanding as a whole, and the expansion degree is not uniform everywhere. For the vorticity field, a two-dimensional Fourier transform is performed to obtain its energy spectrum, which represents the distribution of vortex energy at different scales. If the stirring is intense, the energy is widely distributed at high, medium and low frequencies, and the calculated entropy value of the energy spectrum may be as high as 6.9; if the stirring is slow, a large stable vortex is formed, and the energy is concentrated at low frequencies, and the entropy value may be as low as 2.5.

[0109] In a preferred embodiment, a state space is constructed by the following steps:

[0110] S5.1, collect multiple independent videos of the milk powder stirring process that eventually reach a uniformly mixed state;

[0111] S5.2, extract at least one frame of image from the uniformly mixed stage of each video, and extract a high-dimensional state vector;

[0112] S5.3, collect all the extracted high-dimensional state vectors together, and calculate the mean vector μ and the covariance matrix Σ of the set;

[0113] S5.4, the target uniform state cluster core is the mean vector μ, and the Mahalanobis distance calculation used in state discrimination is based on the inverse matrix of the covariance matrix Σ.

[0114] To calibrate the system, a standard goal state is defined by learning a large number of samples that are known to be well-mixed. By collecting multiple independent mixing videos, the generalization ability of the model is ensured, avoiding overfitting to a particular experimental condition. Only the features from the final uniform stage of each video are extracted, to ensure that the goal state learned is stable and ideal. The extracted high-dimensional state vector contains all the features of the aforementioned spatial texture, temporal color texture, and dynamic flow field, constituting a feature of the mixing state. As an example, 20 mixing videos are collected. From each video, the last 20 seconds of well-mixed uniformity are confirmed, and 5 frames per second are extracted, for a total of 100 frames, so a total of 2000 sample frames are obtained. For each image, a state vector containing 7 feature values, such as spatial contrast, temporal homogeneity, divergence mean, etc., is calculated. A 2000 row by 7 column dataset is obtained. Based on this dataset, a 7-dimensional mean vector μ is calculated, which represents the most typical feature of the well-mixed uniform state. At the same time, a 7 by 7 covariance matrix Σ is also calculated, which describes the normal fluctuation range of each feature value and their correlation in the uniform state. This mean vector and covariance matrix together define the goal uniform region in the high-dimensional feature space.

[0115] In a preferred embodiment, the Mahalanobis distance calculation used in the state discrimination is based on the inverse of the covariance matrix Σ, and specifically includes:

[0116] S6.1, obtain the high-dimensional state vector of the current frame ;

[0117] S6.2, calculate the Mahalanobis distance using the pre-computed mean vector and the inverse of the covariance matrix ; where sqrt() represents the square root operation.

[0118] The high-dimensional state vector of each frame collected in real time is calculated, which represents the mixing state at the current time. The Mahalanobis distance between this vector and the pre-defined goal uniform state center μ is calculated. A small Mahalanobis distance means that the current state is statistically very close to the ideal uniform state. Suppose the state vector calculated for the current frame is [1.1, 7.2, 0.5, 0.6, 2.1, 0.9, 4.8]. Using the pre-computed mean vector μ and the inverse of the covariance matrix The value of the Mahalanobis distance D is calculated by the formula to be 9.7. Two thresholds are preset, for example, the first threshold is 4.0 and the second threshold is 8.0. The two thresholds are obtained by analyzing the Mahalanobis distance distribution of a large number of homogeneous samples and inhomogeneous samples and selecting the limits that can best distinguish them. Since the calculated distance 9.7 is greater than the second threshold 8.0, the current state is determined to be mixed inhomogeneous. If the calculated distance is 3.1, it is less than the first threshold 4.0, and the state will be determined to be mixed homogeneous. If the distance is 6.5, between the two thresholds, the state is determined to be mixed, as shown in Figure 6

[0119] In a second embodiment, a real-time monitoring system for milk powder stirring processing is provided, comprising the following modules:

[0120] The acquisition module is used for acquiring a current frame image of the milk powder material in the stirring container and obtaining a reference frame image before a preset time interval of the current frame image.

[0121] The spatial texture feature calculation module is used for constructing a spatial variant gray level co-occurrence matrix based on the local gradient information of each pixel of the L* channel of the current frame image, and extracting contrast, entropy and energy from the spatial variant gray level co-occurrence matrix as a spatial texture feature set.

[0122] The color texture feature calculation module is used for calculating difference images of each channel of the CIELab space between the current frame image and the reference frame image, extracting gray level co-occurrence matrix features of each difference image, and performing weighted summation on the gray level co-occurrence matrix features by using the information entropy of each channel of the current frame as a weight to obtain a time sequence color texture feature set.

[0123] The flow field feature calculation module is used for calculating a dense optical flow field between the current frame image and the reference frame image, extracting the mean value and variance of the divergence of the optical flow vector field, and the entropy of the energy spectrum of the vorticity, to jointly constitute a dynamic flow field feature set.

[0124] The state judgment module is used for concatenating the spatial texture feature set, the time sequence color texture feature set and the dynamic flow field feature set to obtain a high-dimensional state vector; calculating the Mahalanobis distance of the high-dimensional state vector in a pre-constructed state space and a target homogeneous state cluster core; when the Mahalanobis distance is less than a first threshold, determining that the current stirring state is homogeneous mixing; when the Mahalanobis distance is greater than a second threshold, determining that it is an abnormal state; and when the Mahalanobis distance is between the first threshold and the second threshold, determining that it is mixing.

[0125] In a preferred embodiment, the construction of the spatial variant gray level co-occurrence matrix based on the local gradient information of each pixel of the L* channel of the current frame image and the extraction of contrast, entropy and energy from the spatial variant gray level co-occurrence matrix as a spatial texture feature set comprises: ​

[0126] S2.1, converting the current frame image from RGB space to CIELab space, and extracting the L* channel image;

[0127] S2.2, calculating the gradient amplitude and gradient direction of each pixel point in the L* channel image using the Sobel operator, and quantizing the gradient direction into multiple discrete directions;

[0128] S2.3, for each pixel point in the L* channel image, finding the neighborhood pixels of a preset distance in the quantized gradient direction to form a pixel pair;

[0129] S2.4, constructing a spatial variant gray level co-occurrence matrix according to the L* values of all pixel pairs, and performing normalization processing;

[0130] S2.5, calculating the contrast, entropy and energy based on the normalized spatial variant gray level co-occurrence matrix as the spatial texture feature set.

[0131] In a preferred embodiment, the difference images of the current frame image and the reference frame image in each channel of the CIELab space are calculated, the gray level co-occurrence matrix features of each difference image are extracted, and the information entropy of each channel of the current frame is used as a weight to weight and sum the gray level co-occurrence matrix features, to obtain a time sequence color texture feature set, including:

[0132] S3.1, converting the current frame image and the reference frame image into CIELab space to obtain three channels of L*, a* and b*;

[0133] S3.2, calculating the difference images of the current frame image and the reference frame image for each channel of L*, a* and b* respectively;

[0134] S3.3, calculating the gray level co-occurrence matrix for the three difference images respectively, and extracting the contrast and homogeneity features from each matrix;

[0135] S3.4, calculating the information entropy of each of the L*, a* and b* channels of the current frame image;

[0136] S3.5, using the information entropy of each channel as a weight to cross-channel weight and sum the contrast and homogeneity features extracted in S3.3, to obtain the weighted contrast and homogeneity features, which together constitute the time sequence color texture feature set.

[0137] In a preferred embodiment, the dense optical flow field between the current frame image and the reference frame image is calculated, the divergence mean and variance of the optical flow vector field, and the entropy of the vorticity energy spectrum are extracted, to together constitute a dynamic flow field feature set, including:

[0138] S4.1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel;

[0139] S4.2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field;

[0140] S4.3. Calculate the curl field of the velocity vector field using a difference method;

[0141] S4.4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum;

[0142] S4.5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

[0143] In a preferred embodiment, constructing the state space is achieved by the following steps:

[0144] S5.1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state;

[0145] S5.2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector;

[0146] S5.3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection;

[0147] S5.4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

[0148] Optionally, the Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including:

[0149] S6.1. Obtain the high-dimensional state vector of the current frame ;

[0150] S6.2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

[0151] Those skilled in the art can clearly understand the application by the description of the above embodiments that the application can be implemented by means of software and the necessary universal hardware platforms. Based on such an understanding, the technical solutions of the application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, and the like) execute the methods described in each embodiment or some parts of the embodiments of the application.

[0152] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment. The above-described system and system embodiment are merely illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement it without creative labor.

[0153] The method for providing commodity object information and the electronic device provided by the application are described in detail above, and the principles and implementation manners of the application are described by applying specific examples. The above embodiment is only used to help understand the method of the application and its core idea; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation manner and application range can be changed. In conclusion, the content of the specification should not be understood as a limitation of the application.

Claims

1. A real-time monitoring method for milk powder stirring processing, characterized in that: The following steps are involved: S1. Acquire a current frame image of the milk powder material in the stirring container, and obtain a reference frame image of the current frame image before a preset time interval; S2. For the L* channel of the current frame image, construct a spatial variation gray-level co-occurrence matrix based on the local gradient information of each pixel, and extract contrast, entropy and energy from it as a spatial texture feature set; S3. Calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set; S4, calculating the dense optical flow field between the current frame image and the reference frame image, extracting the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, to form a dynamic flow field feature set; S5. Concatenate the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in the pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that it is mixing.

2. The method according to claim 1, characterized in that The L* channel of the current frame image is constructed based on the local gradient information of each pixel in the spatial variation gray level co-occurrence matrix, and contrast, entropy and energy are extracted as a spatial texture feature set, including: S2.

1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.

2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.

3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.

4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.

5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

3. The method according to claim 1, characterized in that The method comprises calculating the difference images of each channel of the current frame image and the reference frame image in the CIELab space, extracting the gray-level co-occurrence matrix features of each difference image, and using the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set, including: S3.

1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.

2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.

3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.

4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.

5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

4. The method according to claim 1, wherein The dense optical flow field between the current frame image and the reference frame image is calculated, and the divergence mean and variance of the optical flow vector field and the entropy of the curl energy spectrum are extracted to form a dynamic flow field feature set, including: S4.

1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.

2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.

3. Calculate the curl field of the velocity vector field using a difference method; S4.

4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.

5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

5. The method according to claim 1, wherein Constructing the state space is achieved through the following steps: S5.

1. Capture multiple independent videos of the milk powder mixing process until it reaches a uniform state; S5.

2. Extract at least one frame of image from the homogeneous mixing stage of each video segment and extract the high-dimensional state vector; S5.

3. Collect all extracted high-dimensional state vectors together and calculate the mean vector μ and covariance matrix Σ of the collection; S5.

4. The core of the target uniform state cluster is the mean vector μ, and the Mahalanobis distance used in state discrimination is calculated based on the inverse matrix of the covariance matrix Σ.

6. The method according to claim 5, characterized in that The Mahalanobis distance used in the state discrimination is calculated based on the inverse matrix of the covariance matrix Σ, specifically including: S6.

1. Obtain the high-dimensional state vector of the current frame ; S6.

2. Using pre-computed mean vectors and the inverse of the covariance matrix , calculate the Mahalanobis distance ; Among them, sqrt() represents the square root operation.

7. A real-time monitoring system for milk powder mixing processing, characterized in that: Includes the following modules: an acquisition module, configured to acquire a current frame image of the milk powder material in the stirring container and obtain a reference frame image of the current frame image before a preset time interval; A spatial texture feature calculation module is used to construct a spatial variation grayscale co-occurrence matrix based on the local gradient information of each pixel in the L* channel of the current frame image, and extract contrast, entropy and energy from it as a spatial texture feature set; A color and texture feature calculation module is used to calculate the difference image of each channel in the CIELab space between the current frame image and the reference frame image, extract the gray-level co-occurrence matrix features of each difference image, and use the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color and texture feature set; A flow field feature calculation module is used to calculate the dense optical flow field between the current frame image and the reference frame image, extract the divergence mean and variance of the optical flow vector field, and the entropy of the curl energy spectrum, which together constitute a dynamic flow field feature set; The state judgment module is used to cascade the spatial texture feature set, the temporal color texture feature set, and the dynamic flow field feature set to obtain a high-dimensional state vector; calculate the Mahalanobis distance between the high-dimensional state vector and the core of the target uniform state cluster in a pre-constructed state space; when the Mahalanobis distance is less than a first threshold, determine that the current stirring state is uniformly mixed; when the Mahalanobis distance is greater than a second threshold, determine that it is an abnormal state; when the Mahalanobis distance is between the first threshold and the second threshold, determine that mixing is in progress.

8. The system according to claim 7, characterized in that The L* channel of the current frame image is constructed based on the local gradient information of each pixel in the spatial variation gray level co-occurrence matrix, and contrast, entropy and energy are extracted as a spatial texture feature set, including: S2.

1. Convert the current frame image from RGB space to CIELab space, and extract the L* channel image; S2.

2. Calculate the gradient magnitude and gradient direction of each pixel in the L* channel image using the Sobel operator, and quantize the gradient direction into multiple discrete directions; S2.

3. For each pixel in the L* channel image, search for neighboring pixels with a preset distance in the direction of the quantized gradient to form a pixel pair; S2.

4. Construct a spatially variable gray-level co-occurrence matrix based on the L* values ​​of all pixel pairs and perform normalization. S2.

5. Based on the normalized spatial variation gray-level co-occurrence matrix, calculate contrast, entropy and energy as the spatial texture feature set.

9. The system according to claim 7, wherein: The method comprises calculating the difference images of each channel of the current frame image and the reference frame image in the CIELab space, extracting the gray-level co-occurrence matrix features of each difference image, and using the information entropy of each channel of the current frame as a weight to perform weighted summation on the gray-level co-occurrence matrix features to obtain a temporal color texture feature set, including: S3.

1. Convert the current frame image and the reference frame image to CIELab space to obtain three channels: L*, a*, and b*. S3.

2. For each channel of L*, a*, and b*, calculate the difference image between the current frame image and the reference frame image; S3.

3. Calculate the gray-level co-occurrence matrix for each of the three differential images and extract contrast and homogeneity features from each matrix; S3.

4. Calculate the information entropy of each of the three channels L*, a*, and b* of the current frame image; S3.

5. Using the information entropy of each channel as a weight, perform cross-channel weighted summation on the contrast and homogeneity features extracted in S3.3 to obtain weighted contrast and homogeneity features, which together constitute the temporal color texture feature set.

10. The system according to claim 7, wherein: The dense optical flow field between the current frame image and the reference frame image is calculated, and the divergence mean and variance of the optical flow vector field and the entropy of the curl energy spectrum are extracted to form a dynamic flow field feature set, including: S4.

1. Calculate the optical flow field between the current frame image and the reference frame image using a dense optical flow estimation algorithm to obtain a velocity vector field for each pixel; S4.

2. Calculate the divergence field of the velocity vector field using a difference method, and calculate the mean and variance of the divergence field; S4.

3. Calculate the curl field of the velocity vector field using a difference method; S4.

4. Perform a two-dimensional Fourier transform on the curl field and calculate an energy spectrum; S4.

5. Normalize the energy spectrum to a probability distribution and calculate the information entropy as the entropy of the curl energy spectrum.

Images