Soft measurement method of liquid strain biomass based on machine vision

Abstract

The invention provides a soft measurement method for liquid strain biomass based on machine vision. The soft measurement method comprises the following steps: off-line sampling and analyzing liquid strain biomass parameters and evolution rules thereof in a full fermentation period; a biomass parameter soft measurement model M1 is established according to the manually measured auxiliary variable characteristics M-Feature; the method comprises the following steps: collecting visible light images and visible-near-infrared I region-near-infrared II region multi-source spectrum data of liquid strains in different fermentation stages, constructing a spectrum data set, and constructing a multi-task machine vision model V1 for extracting machine vision-based auxiliary variable features E-Features; the biomass parameter soft measurement model M1 and the multi-task machine vision model V1 are combined, correlation between E-Features and M-Features is established, and a cascade machine vision soft measurement model MV is obtained; and deploying the MV model to a terminal embedded intelligent device, and designing interaction software for task execution. According to the method, the trend of biomass parameter evolution in the liquid strain fermentation process can be evaluated on line in real time so as to guide production and application of strains.

Description

Technical Field

[0001] The invention relates to the technical field of soft measurement of biomass parameters of fermentation engineering, and in particular to a method for detecting biomass parameters in a liquid strain fermentation process by using cascade machine vision. Background Art

[0002] The quality and quantity of liquid strains determine the quality and yield of edible fungi. In order to obtain target strains with high biomass, high purity and good vitality in the shortest time, it is necessary to conduct in-situ real-time non-contact monitoring of the dynamic evolution of biomass during the fermentation process to obtain stable and high-quality output. The fermentation process of liquid strains is usually a dynamic growth and evolution process, involving the fission growth of strains, the nutrient consumption of fermentation liquid and the accumulation of fermentation products. Therefore, its fermentation process is a complex nonlinear system with large hysteresis, multivariate and strong coupling, and the internal reaction mechanism is very complex. In the actual fermentation production process, the parameters that can be automatically detected are mainly concentrated on some physical and chemical parameters that can be directly and easily measured by existing sensors, such as temperature, pH, dissolved oxygen, tank pressure, etc., while other important key biochemical parameters that affect the quality of the fermentation process, such as substrate concentration, bacterial concentration and product concentration, must be obtained by offline analysis methods.

[0003] Offline analysis has a large lag and is prone to fermentation contamination, thus affecting fermentation efficiency and product quality. Soft measurement technology is one of the effective ways to solve the above problems. Soft measurement technology is an indirect measurement method that uses easily measured parameters (auxiliary variables) to achieve the derivation and measurement of key parameters (dominant variables) that are not easy to measure through modeling. The core of soft measurement technology is to establish a soft measurement model that characterizes the mathematical relationship between auxiliary variables and dominant variables. Therefore, it is generally necessary to solve the mathematical modeling problem of soft measurement models in order to achieve a robust characterization of dominant variables. At present, there are three main soft measurement modeling methods: soft measurement based on mechanism analysis (white box model), soft measurement based on neural network or deep learning (black box model), and soft measurement based on a hybrid model of mechanism analysis and machine learning (gray box model).

[0004] Regardless of whether the white box, black box or gray box soft sensor model is established, it all relies on the acquisition of one-dimensional sequence physical and chemical parameters, such as physical and chemical parameters that are easy to measure directly, such as temperature, pH, dissolved oxygen and tank pressure, and then the corresponding mechanism modeling or big data analysis is carried out. Since these one-dimensional sequence physical and chemical parameters are external environmental parameters of the fermentation process, or process parameters that indirectly reflect biological parameters, they cannot robustly characterize the biological parameters of the fermentation process. A large number of studies have shown that the use of visible-infrared spectral data can achieve online real-time monitoring of biological parameters, but for the online monitoring of the biomass of edible fungi liquid strains, there are the following difficulties, such as: the liquid strains are always in the process of pneumatic stirring, which affects the stable acquisition of the mycelium spectral characteristics; the strong absorption of water in the near-infrared band will affect the robustness and accuracy of the measurement of biological parameters; there is a contradiction between the demand for online real-time monitoring of biomass parameters in the fermentation process and the accuracy and lightweight of the model edge deployment. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method for detecting the biomass of liquid strains based on machine vision, a soft measurement scheme for inferring parameters that are difficult to measure by a cascaded machine vision model that integrates multi-source graph information, namely a "color box" model, to make up for the limitations of the "white box" based on mechanism analysis, the "black box" based on neural network or big data analysis, and the "gray box" scheme based on a hybrid model of the above two, so as to realize online real-time visual monitoring of biomass parameters in the fermentation process of liquid strains, so as to guide the production and application of liquid strains in the fermentation process.

[0006] To achieve the above object, the technical solution of the present invention is: a soft measurement method of liquid bacterial biomass based on machine vision comprises the following steps:

[0007] S1: Analyze the dominant variables of liquid strain biomass parameters and their evolution during the whole fermentation cycle. The dominant variables of biomass parameters Bio-Indices include: strain concentration %, strain fresh weight g / L and strain dry weight g / L;

[0008] S2: Offline sampling and manual measurement of auxiliary variable features M-Features of associated liquid strain biomass parameters, including: fermentation liquid volume V ls , strain volume V sp , strain quality M sp And the fermentation liquid quality M ls , establish the soft sensor model M1 of the dominant variable of biomass parameters based on auxiliary variable features M-Features;

[0009] S3: Collect visible-near infrared I-near infrared II hyperspectral data of liquid bacteria at different fermentation stages to construct a multi-source atlas dataset characterizing the dynamic evolution of liquid bacteria biomass;

[0010] S4: Build a multi-task machine vision model V1 based on a multi-source graph dataset, and extract key graph features to obtain auxiliary variable features E-Features based on machine vision, including: fermentation window area S ls , strain area S sp , Bacteria Spectral Index I sp and fermentation broth spectral index I ls .

[0011] S5: Combining the biomass parameter soft measurement model M1 and the multi-task machine vision model V1, a cascade machine vision soft measurement model MV is obtained to perform soft measurement analysis of Bio-Indices;

[0012] S6: deploy the cascaded machine vision soft measurement model MV on the terminal embedded intelligent device;

[0013] S7: Design interactive software for machine vision task execution, realize cross-scenario transfer learning and online real-time reasoning of cascaded machine vision soft measurement model MV, cross-scenario transfer learning and cross-scenario application.

[0014] Preferably, the S1 is specifically:

[0015] Offline sampling and analysis of liquid strains specifically include: aseptic sampling, filtration, pure water washing, centrifugation, freeze drying, laboratory analysis and measurement of strain biomass parameters; the full fermentation cycle includes: adaptation period (1-2 days), rapid growth period (3-6 days), maturity period (7-8 days), aging period (8-10 days); the evolution law of the biomass parameter leading variable Bio-Indices refers to the strain concentration C on (%), fresh weight of strains BM f (g / L) and strain dry weight BM d (g / L) The dynamic changes of the fermentation content as the fermentation time progresses during the entire fermentation cycle.

[0016] Preferably, S2 is specifically:

[0017] The volume of fermentation liquid V in the auxiliary variable feature M-Features ls , is the volume of the fermentation liquid of the sample, and the sampling volume can be 400mL; the strain volume V in the auxiliary variable feature M-Features sp , is the total volume of all bacterial clusters in a certain volume of sample; the bacterial mass M in the auxiliary variable feature M-Features sp, is the total mass of all bacterial clusters in a certain volume of sample. The fermentation broth mass M in the auxiliary variable feature M-Features ls , refers to the quality of the fermentation broth of the sample;

[0018] Establish a soft sensor model M1, and realize the inference of the biomass parameter dominant variable Bio-Indices by establishing a mathematical equation between the auxiliary variable and the dominant variable; the process of establishing the mathematical equation depends on the mechanism analysis and mathematical derivation between the auxiliary variable characteristics and the dominant variable characteristics. The specific process is: the bacterial concentration C in the Bio-Indices on Depends on the volume V of the bacteria in M-Features sp The fermentation liquid volume V ls ; The fresh weight of strains in the Bio-Indices BM f Depends on the quality of the strain M in M-Features sp ; Dry weight of strains in Bio-Indices BM d Depends on the fresh weight of the strain BM f and moisture content of bacterial mass, the simultaneous equations are as follows:

[0019]

[0020] In the formula,

[0021] a is the volume of the bacteria V sp The correlation coefficient between offline analysis and online measurement is in the range of (1.0-1.2);

[0022] b is the volume of fermentation liquid V ls The correlation coefficient between offline analysis and online measurement is not significantly affected by pressure changes and is generally 1.0;

[0023] c is the mass of the strain M sp The correlation coefficient between offline analysis and online measurement is not significantly affected by pressure changes and is generally 1.0;

[0024] d is the moisture content coefficient of the bacterial mass, which is linearly adjusted according to the liquid bacterial species and fermentation cycle, and the value range is (8.0-9.5).

[0025] Preferably, S3 is specifically:

[0026] A multi-source atlas data set is established, including visible light RGB two-dimensional images and visible light-near infrared I zone-near infrared II zone one-dimensional spectral data; the visible light band range is 380-780nm, the near infrared I zone band range is 780-1000nm, the near infrared II zone band range is 1000-1700nm, and the band resolution is 1nm; wherein, the selection of the visible light band can characterize the color change characteristics of the strain and the fermentation liquid color, the selection of the near infrared I zone and II zone bands for transmission can characterize the body compactness of the bacterial mass and the evolution characteristics of the surface burrs during the fermentation process, and the selection of the near infrared I zone and II zone bands for diffuse reflection can characterize the evolution of the fermentation products and the fermentation substrate of the fermentation liquid from a heterogeneous suspension, a turbid liquid, a liquid sol to a homogeneous true solution as the fermentation progresses, thereby obtaining the corresponding spectral index characteristics.

[0027] Preferably, S4 is specifically:

[0028] The multi-task machine vision model V1 is constructed by cascading a multi-task machine vision model to realize the extraction and analysis of auxiliary variable features; the multi-task machine vision model performs a series of visual tasks respectively, including classification tasks, target detection tasks, semantic segmentation tasks and morphological image processing tasks; the classification task realizes the task of classifying the liquid strain fermentation tank and other equipment; the target detection task realizes the task of identifying and locating the fermentation window position of the fermentation tank; the semantic segmentation task realizes the task of segmenting the liquid strain area in the fermentation window; the morphological image processing task realizes the separation of strains and fermentation liquid, and extracts key spectral features, including LBP (local binary pattern) features, HOG (histogram of oriented gradients) features, roundness factors, morphological factor features, grayscale, and spectral index features; thereby deriving auxiliary variable features E-Features based on machine vision including strain area, fermentation window area, strain and fermentation liquid spectral index.

[0029] Preferably, S5 is specifically:

[0030] The auxiliary variable features E-Features obtained by the multi-task machine vision model V1 are associated with the auxiliary variable features M-Features measured manually, specifically: the volume of the strain V in M-Features sp The fermentation window area S observed in E-Features ls Inner bacterial area S sp Mathematical equations can be established; the volume of fermentation liquid V in M-Features ls The volume of the fermentation broth sample; M in M-Features is the mass of the strain sp The area of ​​bacteria in E-Features sp and bacterial species spectral index Isp Mathematical equations can be established; the fermentation liquid quality M in M-Features ls and fermentation broth spectral index I ls And fermentation window area S ls Mathematical equations can be established; the mathematical equations between the above E-Features and M-Features are combined as follows:

[0031]

[0032] Where e is the bacterial area S measured online sp The corresponding bacterial volume V sp A constant correction factor between ;

[0033] f(I sp ) is the correlation function between the online measurement of bacterial species spectral index and bacterial species density;

[0034] f(I ls ) is the correlation function between the online measurement of the fermentation liquid spectral index and the fermentation liquid density;

[0035] g is the fermentation window area S ls The corresponding fermentation liquid volume V ls A constant correction factor between ;

[0036] The M-Features derived from E-Features are input into the biomass parameter soft measurement model M1 to derive the biomass parameter Bio-Indices; thereby achieving indirect measurement of the leading variables of the biomass parameters by extracting auxiliary variable features through machine vision.

[0037] Preferably, the terminal embedded intelligent device in the S6 includes: an edge computing device with an independent graphics card, a built-in wireless network card, an SD card slot for data expansion, and an anti-seismic, lightning-proof and anti-static design for industrial-grade scenario applications. The model deployment process includes: adopting an unstructured pruning strategy to perform lightweight improvements on the machine vision model.

[0038] Preferably, the interactive software for executing the machine vision model task in S7 is a human-computer interface designed by pyQT6 to realize the function of interactively executing the machine vision task; the functions of the interactive software for executing the machine vision task include: on the human-computer interaction interface, realizing database expansion, model mobilization, model execution, model convergence training, model preservation and model reasoning and biomass parameter measurement effect display; the cross-scene transfer learning of the cascaded machine vision soft measurement model MV refers to the generalization migration of the model through real-time interactive establishment of data sets, interactive model loading, convergence training under cross-scene conditions; the cross-scene application refers to the target biomass parameter estimation under different strains and different field environments. If the estimated biomass parameter dominant variable does not meet the accuracy requirement, the data set will continue to be expanded, and the model loading and convergence training will be repeated until the accuracy requirement is met; if the estimated biomass parameter has met the accuracy requirement, it means that the model meets the online real-time non-invasive monitoring of the biomass parameter of the current scene.

[0039] Compared with the prior art, the present invention has the following beneficial effects: a soft measurement method for liquid strain biomass parameters based on cascade machine vision is provided to achieve online non-invasive monitoring of biological parameters that are difficult to measure directly. This is achieved by integrating the "color box model" of multi-source maps and machine learning, that is, the cascade machine vision soft measurement model, to achieve the inference of parameters that are difficult to measure. The auxiliary variable features E-Features are obtained through machine vision, associated with M-Features, and then input into the soft measurement model to perform indirect measurement of the biomass parameter leading variable Bio-Indices. Since the "color box" soft measurement method has both the mechanism analysis of two-dimensional maps and the feature extraction capabilities of neural networks, it can make the inference effect more explanatory and robust than the feature extraction capabilities usually based only on neural network learning, or the soft measurement model based only on one-dimensional environmental parameters, thereby providing an ideal solution for non-invasive real-time monitoring of biomass parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Attached Figure 1 The flowchart of the biomass parameter analysis based on offline sampling of the preferred embodiment of the present invention; wherein, step a represents aseptic sampling, step b represents molecular sieve filtration, step c represents the strain after filtration and pure water cleaning, step d represents the volume measurement of the fermentation liquid after filtration, step e represents the fresh weight (g / L) of the strain obtained by offline analysis and measurement of the biomass parameter, step f represents the strain concentration (%) of the biomass parameter obtained by offline analysis and measurement, step g represents freeze-drying the strain obtained in step c, and step h represents the dry weight (g / L) of the strain obtained by offline analysis and measurement of the biomass parameter;

[0041] Attached Figure 2This is a diagram of the connection and components of the machine vision platform of the preferred embodiment of the present invention; wherein, (a) WIFI antenna component, (b) 4020 fan, (c) radiator, (d) JetsonNano module, (e) Jetson Nano carrier board, (f) 128G microSD card, (g) portable monitoring screen, (h) 2-DOF gimbal, (i) CSI camera;

[0042] Attached Figure 3 The experimental results of the classification task in the multi-task machine vision of the present invention; wherein (a) represents the loss and evaluation index of the classification task training process, (b) represents the confusion matrix of the training result, and (c) represents the classification task training effect;

[0043] Attached Figure 4 The experimental results of the target detection task in the multi-task machine vision of the present invention; wherein (a) represents the loss and evaluation index of the target detection task training process, (b) represents the confusion matrix of the training result, (c) represents the correlation diagram between the label and the bounding box of the target detection task training process, and (d) represents the training result of the target detection task;

[0044] Attached Figure 5 The experimental results of the semantic segmentation task in the multi-task machine vision of the present invention; wherein (a) represents the loss and evaluation index of the semantic segmentation task training process, (b) represents the confusion matrix of the training result, (c) represents the correlation diagram between the label and the bounding box of the semantic segmentation task training process, and (d) represents the training result of the object detection task;

[0045] Attached Figure 6 The experimental effect diagram of the machine vision task of the present invention; wherein step a represents the fermentation tank window area, step b represents the liquid strain target area segmentation, and step c represents the liquid strain target area of ​​interest extraction;

[0046] Attached Figure 7 The experimental results of the biomass parameter inference based on machine vision are preferred in the present invention; wherein a represents the machine vision processing process of the fermentation tank during the whole fermentation cycle, b represents the dynamic curve diagram of biomass change obtained based on machine vision processing, c represents the correlation coefficient between the biomass parameter extracted based on machine vision and the biomass parameter based on manual measurement, and d represents the dynamic curve diagram of each biomass parameter change after correction by the correlation coefficient;

[0047] Attached Figure 8 The interactive software interface for performing independent machine vision tasks of the present invention; wherein (a) represents the classification task interface, (b) represents the target detection task interface, and (c) represents the semantic segmentation task interface;

[0048] Attached Fig. 9An interactive software interface for performing machine vision synthesis tasks for the present invention;

[0049] Attached Fig.10 The figure is a comparison between the biomass parameters inferred by machine vision and those based on manual measurement in the present invention; wherein a is the biomass parameter measured manually, b is the biomass parameter inferred by machine vision, c and d are the normalized processing results of the biomass parameters, e and f are the Boltzmann fitting results of the biomass growth curve of the strain, and g and h represent the corresponding relationship between the biomass parameters inferred by machine vision and those based on manual measurement and their Pearson correlation;

[0050] Fig.11 This is a diagram illustrating the process of inferring biomass parameters based on machine vision of the present invention, wherein step a represents the establishment of a data set, step b represents the establishment of a machine vision model, step c represents the construction of a machine vision platform, step d represents the result of execution based on cascaded machine vision, step e represents the establishment of reasoning association equations for various biomass parameters, and step f represents the interactive interface for executing biomass parameter inference;

[0051] Fig.12 It is a flow chart of the cascaded machine vision model MV of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0055] refer to Figures 1 to 12 ,The soft measurement of biomass parameters in the fermentation process of velvet antler mushroom liquid spawn based on machine vision includes the following steps:

[0056] S1: Offline sampling and analysis of the evolution of the leading variables of liquid strain biomass parameters during the whole fermentation cycle; measuring the liquid strain biomass indicators of Pleurotus eryngii at different stages during the whole fermentation cycle, wherein the whole fermentation cycle includes an adaptation period (2 days), a rapid growth period (3 days), a maturity period (2 days), and a senescence period (2 days); further, the offline sampling and analysis includes: aseptic sampling, alcohol disinfection, flame isolation, and sampling; constant temperature centrifugation: 1500r / min, 15min; filtration, 150 mesh molecular sieve; cleaning, RO pure water cleaning, three times; freeze drying, -80 refrigerator freezing hard, -50℃ freeze drying to room temperature; measuring the leading variable indicators of biomass parameters, recording experimental data and archiving; offline sampling and analysis process such as Figure 1 As shown;

[0057] Furthermore, the biomass parameter leading variable indicators of the offline sampling analysis include the strain concentration C on (%), fresh weight of strains BM f (g / L) and strain dry weight BM d (g / L); further, the biomass parameter calculation equation is:

[0058]

[0059]

[0060] Where V ls Indicates the volume of fermentation broth in the sample taken;

[0061] V sp Table of the volume of bacteria in the sampled samples;

[0062] Indicates the mass of water reduced by freeze drying.

[0063] S2: Offline sampling and manual measurement of auxiliary variable features M-Features associated with liquid strain biomass parameters, including liquid strain fermentation liquid sampling volume V ls , the volume of the bacterial strain V sp , strain quality M sp And the fermentation liquid quality M ls Parameters; Establish the soft sensor model M1 of the biomass parameter dominant variable Bio-Indices;

[0064] Auxiliary variable features M-Features V ls Refers to the volume of fermentation broth of the sampled sample. The sampling volume of each sample can be divided into 400 mL. V sp, refers to the total volume of all bacterial clusters in a certain volume of sample; M in the auxiliary variable feature M-Features sp , refers to the total mass of all bacterial clusters in a certain volume of sample. The M in the auxiliary variable feature M-Features ls , refers to the mass of the fermentation liquid after removing the mass of the strain from a certain volume of sample;

[0065] The soft measurement model M1 is an indirect measurement method, which realizes the inference of the biomass parameter dominant variable Bio-Indices by establishing mathematical equations of auxiliary variables and dominant variables; the M1 establishment process can be obtained by performing mechanism analysis and mathematical deduction between the auxiliary variable characteristics and the dominant variable characteristics;

[0066] Further, the mechanism analysis specifically refers to: fresh weight of strain BM f Depends on the quality of the strain M sp , and then corrected by a constant coefficient; bacterial concentration C on Depends on the volume of the culture V sp The fermentation liquid volume V ls , can be inferred through the three-dimensional volume of M-Features or the two-dimensional area of ​​E-Features; and the dry weight of the strain BM d Depends on the quality of the strain M sp and the moisture content of bacterial mass, which can be obtained through statistical analysis;

[0067] Furthermore, the specific process of the mathematical derivation is as follows: the concentration of the bacterial species C in the Bio-Indices on Depends on the volume V of the bacteria in M-Features sp The fermentation liquid volume V ls ; The fresh weight of strains in the Bio-Indices BM f Depends on the quality of the strain M in M-Features sp ; Dry weight of strains in Bio-Indices BM d Depends on the fresh weight of the strain BM f and moisture content of bacterial mass, the simultaneous equations are as follows:

[0068]

[0069] In the formula,

[0070] a is the volume of the bacteria V sp The correlation coefficient between offline analysis and online measurement is in the range of (1.0-1.2);

[0071] b is the volume of fermentation liquid V lsThe correlation coefficient between the offline analysis and the online measurement does not take into account the pressure change and is generally taken as 1.0;

[0072] c is the mass of the strain M sp The correlation coefficient between the offline analysis and the online measurement does not take into account the pressure change and is generally taken as 1.0;

[0073] d is the moisture content coefficient of the bacterial mass, which is linearly adjusted according to the liquid bacterial species and fermentation cycle, and the value range is (8.0-9.5).

[0074] S3: Collect visible light images and near-infrared multispectral reflectance images of liquid bacteria at different fermentation stages to construct a multi-source atlas dataset that characterizes the dynamic evolution of liquid bacteria biomass; use a hyperspectral imager covering the visible-near infrared I-near infrared II (380-1700nm) band range to collect spectral information of biomass indicators at different fermentation stages and establish a multi-source atlas dataset;

[0075] Furthermore, the multi-source atlas data set includes visible light RGB two-dimensional images, visible light-near infrared I zone-near infrared II zone hyperspectral one-dimensional spectrum data; the visible light band range is 380-780nm, the near infrared I zone band range is 780-1000nm, the near infrared II zone band range is 1000-1700nm, and the band resolution is 1nm; wherein, the selection of the visible light band can characterize the change characteristics of the color of the strain and the color of the fermentation liquid, and the selection of 580-595nm in the visible light band can characterize the change characteristics of the color of the strain and the color of the fermentation liquid;

[0076] Furthermore, the selection of transmission bands of the near-infrared zone I and zone II hyperspectral spectrum can characterize the evolution characteristics of the body compactness and machine surface burrs of the bacterial mass during the fermentation process; the selection of diffuse reflection bands of the near-infrared zone I and zone II hyperspectral spectrum can characterize the evolution characteristics of the colloidal dispersion system of the fermentation liquid, the fermentation products and the fermentation substrate, such as the evolution of the fermentation products and the fermentation substrate of the fermentation liquid from a heterogeneous suspension, a turbid liquid, a liquid sol to a homogeneous true solution as the fermentation progresses, thereby obtaining corresponding spectral index characteristics.

[0077] S4: Building on multi-source atlas datasets Multi-task machine vision modelV1 is used to extract auxiliary variable features E-Features and build a multi-task machine vision model V1 based on deep learning. It is a machine vision model that cascades multiple tasks to achieve the extraction and analysis of key graph features. The selection of the multi-task machine vision model is to enable the machine vision to be positioned in the key window area for analysis, reduce the amount of calculation and improve the accuracy. The specific process is first to use the multi-task machine vision model to enable the vision model to identify and locate the window position, that is, find the area of ​​interest, and then perform key target detection and analysis in the area of ​​interest; the segmentation task of the cascade task is to detect target objects such as target bacteria and fermentation liquid in the fermenter based on the graph data obtained by the target detection task in the previous stage. This example uses threshold segmentation and morphological segmentation for processing to obtain image information of key targets.

[0078] Furthermore, the construction of a multi-task machine vision model is based on YOLOv8 as a basic model, and implements multi-task machine vision tasks to achieve the extraction and analysis of key atlas information;

[0079] Furthermore, the multi-task machine vision model performs a series of visual tasks, including classification tasks, object detection tasks, semantic segmentation tasks and morphological image processing tasks;

[0080] Furthermore, the classification task is to realize the classification of liquid strain fermentation tanks and other equipment; the training set inference result of the model classification task is as follows: Figure 3 ;in Figure 3 (a) represents the loss and evaluation index of the classification task training process, Figure 3 (b) represents the confusion matrix of the training results. Figure 3 (c) indicates the training effect of classification task;

[0081] Furthermore, the target detection task is to identify and locate the fermentation window position of the fermentation tank; the training set reasoning result of the target detection task is as follows: Figure 4 ;in Figure 4 (a) represents the loss and evaluation index of the target detection task training process, Figure 4 (b) represents the confusion matrix of the training results. Figure 4 (c) Correlation diagram of labels and bounding boxes during the object detection task training process. Figure 4 (d) represents the training results of the target detection task;

[0082] Furthermore, the semantic segmentation task refers to the task of segmenting the liquid strain area in the fermentation window; the training set reasoning result of the semantic segmentation task is as follows: Figure 5 ;in Figure 5(a) represents the loss and evaluation index of the semantic segmentation task training process. Figure 5 (b) represents the confusion matrix of the training results. Figure 5 (c) Correlation diagram of labels and bounding boxes during the semantic segmentation task training process. Figure 5 (d) represents the training results of the target detection task;

[0083] Furthermore, the morphological image processing task is to separate the strains from the fermentation liquid and to extract and analyze the key features of the image. The process and results of morphological image processing are as follows: Figure 6 ;in, Figure 6 Step a represents the fermentation tank window area, Figure 6 Step b represents the segmentation of the target area of ​​the liquid bacteria. Figure 6 Step c represents the extraction of the target region of interest of the liquid bacterial strain;

[0084] Furthermore, the threshold segmentation adopts an adaptive threshold segmentation method, which is a kind of local threshold segmentation. Unlike the global threshold, a threshold is not used for the entire matrix, but a corresponding threshold is used for each position of the input matrix. Assuming that the input image is I, the steps of the adaptive threshold segmentation algorithm are as follows:

[0085] Thresh=(1-ratio)*f smooth (I)

[0086]

[0087] Among them, Thresh represents the adaptive threshold matrix, f smooth (I) represents the result of smoothing the image, ratio is the operation ratio, O(r,c) represents the pixel value of a point in the image after processing, where r represents the row vector of the image and c represents the column vector of the image. This formula shows that if the pixel value I(r,c) of a point in the image is greater than the threshold Thresh(r,c) at that point, the pixel value of the point is 255, otherwise it is 0;

[0088] Furthermore, the morphological image processing task adopts a closing operation, and the noise and frame adhesion problems existing in the threshold segmentation image are solved by processing methods such as corrosion and expansion; the corrosion operation can eliminate the boundary of the target object, resulting in the target image being one pixel smaller than the original target, which can separate the originally adhered graphics, but reduce the size of the original graphics; the effect of the expansion operation is just the opposite, and two adhered graphics can be merged; for the multispectral image of the liquid bacteria, a circular structural element is selected for the closing operation, which can meet the requirements of filling the small holes and cracks in the image without changing the overall position and shape of the graphics, thereby improving the detection rate of bacterial clusters and obtaining more accurate key atlas features;

[0089] Furthermore, the key atlas features include: LBP features, HOG features, morphological parameter features (morphological factor, roundness factor, area perimeter factor), spectral index features (grayscale, average grayscale), so as to derive auxiliary variable features E-Features based on machine vision including strain area, fermentation window area, strain and fermentation liquid spectral index;

[0090] Furthermore, the derivation process between the key spectral features and E-Features is polynomial response surface analysis-polynomial regression analysis; the response surface analysis method is used to analyze the changes of the target variable E-Features when the key spectral features change consistently or inconsistently, and the E-Features regression equation can be established by weight allocation combined with multivariate statistical analysis;

[0091] Furthermore, the polynomial regression analysis is specifically implemented by least squares fitting to find a suitable approximate function, using the significance test of regression analysis to understand the strength of the relationship between the independent variable and the response variable, and to test whether the fitting model is appropriate; first, using the low-order polynomial approximation of the independent variable in some range, that is, the first-order regression model, when the experimental area is close to the optimal response value, the curvature of the real response surface will increase, and the second-order model is considered; secondly, if the second-order regression model is well fitted, the second-order model can be used to obtain the optimal combination point and the characteristic response surface; furthermore, if the second-order model is still lacking in adaptability when fitting, the local optimal operating state can be obtained or a higher regression model can be further fitted, such as a cubic or quartic multi-order model; the first-order equation model adopted in this embodiment is:

[0092]

[0093] Where: is the response target of the first-order model, and the parameter x i (i∈(1,k)) are the independent variables, is the correlation coefficient.

[0094] The second-order equation model is:

[0095]

[0096] Where: is the response target of the second-order model, and the parameter x i (i∈(1,k)), x j (j∈(i,k)) are the independent variables, is the correlation coefficient.

[0097] The third-order equation model is:

[0098]

[0099] Where: is the response target of the three-world model, The parameters x are i The upper and lower bounds of the range, are the correlation coefficients.

[0100] S5: Combine the biomass parameter soft measurement model M1 and the multi-task machine vision model V1, establish the association between the auxiliary variable features E-Features obtained by machine vision and the auxiliary variable features M-Features obtained by manual measurement, and obtain the soft measurement model MV for biomass parameter measurement based on cascade machine vision, such as Figure 5 As shown;

[0101] Furthermore, the correlation equation between E-Features and M-Features is corrected and correlated by introducing weight coefficients through mechanism analysis and data statistics. The specific process is: the volume of bacteria in M-Features V sp The fermentation window area S observed in E-Features ls Inner bacterial area S sp Mathematical equations can be established; the volume of fermentation liquid V in M-Features ls The volume of the fermentation broth sample; M in M-Features is the mass of the strain sp The area of ​​bacteria in E-Features sp and bacterial species spectral index I sp Mathematical equations can be established; the fermentation liquid quality M in M-Features ls and fermentation broth spectral index I ls And fermentation window area S ls Mathematical equations can be established; the mathematical equations between the above E-Features and M-Features are combined as follows:

[0102]

[0103] Where e is the bacterial area S measured online sp The corresponding bacterial volume V sp A constant correction factor between ;

[0104] f(I sp ) is the correlation function between the online measurement of bacterial species spectral index and bacterial species density;

[0105] f(I ls ) is the correlation function between the online measurement of the fermentation liquid spectral index and the fermentation liquid density;

[0106] g is the fermentation window area S ls The corresponding fermentation liquid volume V ls A constant correction factor between ;

[0107] The M-Features derived from E-Features are input into the biomass parameter soft measurement model M1 to derive the biomass parameter Bio-Indices; by training the cascade machine vision soft measurement model MV, the association between the multi-source atlas dataset and the auxiliary variable features E-Features is established to infer the biomass parameters; the experimental and inference results of the liquid strain biomass parameters based on machine vision are shown in Figure 7 As shown, Figure 7 Step a represents the machine vision processing process of the fermentation tank during the whole fermentation cycle. Figure 7 Step b represents the dynamic curve diagram of biomass change obtained based on machine vision processing. Figure 7 Step c represents obtaining the correlation coefficient between the biomass parameter extracted based on machine vision and the biomass parameter based on manual measurement, Figure 7 Step d represents the dynamic curve diagram of each biomass parameter change obtained after correlation coefficient correction;

[0108] S6: Deploy the machine vision model MV on an embedded edge computing device; the embedded edge computing device is NIVIDA's JETSON Nano, including: a CUDA128 core computing unit, a built-in wireless network card, a 64G SD card slot for data expansion, and an IPX4 design for earthquake resistance, lightning protection, and anti-static protection for industrial-grade scene applications. The embedded edge computing device platform has the following effects: Figure 2 ,in, Figure 2 (a) represents its WIFI antenna component, Figure 2(b) represents its 4020 fan, (c) represents its radiator, (d) represents its Jetson Nano module, (e) represents its Jetson Nano carrier board, (f) represents its 128G microSD card, (g) represents its portable monitoring screen, (h) represents its 2-DOF gimbal, and (i) represents its CSI camera; lightweight deployment of the model includes: adopting an unstructured pruning strategy to improve the lightweight machine vision model; research and development of edge computing equipment, including multi-source map information acquisition control, image preprocessing and deep learning models, and deploying intelligent edge computing devices on terminals. The intelligent edge computing embedded device selected in this preferred example is NVDIA's Jetson Nano 4GB, which uses a quad-core 64-bit ARM central processing unit (CPU) and a 128-core integrated NVDIA graphics card (GPU), which can provide 472GFLOPS of computing performance. Equipped with the Ubuntu operating system and a rich CUDA toolkit, it is fully compatible with mainstream open source machine learning frameworks such as Pytorch and TensorFlow, and computer vision processing tool libraries such as OpenCV and ROS, making it easy to deploy AI-based machine learning reasoning work on the terminal.

[0109] S7: Design the interactive software for the execution of the machine vision task, realize the transfer learning and online real-time reasoning of the cascade model, cross-scenario transfer learning and cross-scenario application. The interactive software for the execution of the machine vision model task is a human-computer interaction interface for the execution of the machine vision task designed by pyQT6, such as Figure 8 and Fig. 9 ; Figure 8 (a) represents the classification task interface. Figure 8 (b) represents the target detection task interface. Figure 8 (c) represents the semantic segmentation task interface, and Fig. 9 The invention provides a comprehensive interface for interactive software for executing machine vision tasks; the model transfer learning is through interactive establishment of data sets, interactive model loading and convergence training, and real-time model reasoning effect evaluation. The functions of the interactive software for executing machine vision tasks include: on the human-computer interaction interface, realizing database expansion, model mobilization, model execution, model convergence training, model preservation, model reasoning, and biomass parameter soft measurement effect display.

[0110] Furthermore, the offline sampling and analysis of biomass parameters were compared with the online analysis results of machine vision. Fig.10 (a) with Fig.10 (b) There are three biomass parameters: concentration, fresh weight and dry weight. Since the ranges of the three values ​​vary greatly, the measurement results of the three are normalized, such as Fig.10(c). It can be seen that the evolution trends of the three parameters are the same, and the normalized parameter results obtained by machine vision analysis are basically the same. This is because the three parameters are related by first-order linear relationship in machine vision processing, so the normalized results of the three biomass parameters are also basically similar. The statistical analysis between the measurement results of biomass parameters based on offline analysis and machine vision shows that r 2 is 0.963, RMSE is 0.027, and the calculated results of Pearson correlation coefficients between the parameters are 0.91-0.97, thus proving that the biomass parameters based on machine vision reasoning and those based on offline sampling analysis are strongly correlated.

[0111] In summary, the soft measurement method of liquid strain biomass parameters based on cascade machine vision provided by the present invention is a "color box" method that integrates multi-source maps and machine vision, that is, the auxiliary features E-Features of the maps obtained by cascade machine vision are associated with the auxiliary features M-Features of manual measurement, and then input into the soft measurement model, and finally the indirect measurement of the biomass parameters Bio-Indices of the liquid strain fermentation process is realized. Since the "color box" soft measurement method has both the mechanism analysis of the map and the feature extraction capabilities of the neural network, it integrates the auxiliary features of the map of cascade machine vision and the auxiliary features of manual measurement to infer the biomass parameters, and the inference results are more explanatory and robust. Finally, the above-mentioned machine vision model is deployed on the terminal for visual online monitoring, to realize non-invasive real-time monitoring of biomass parameters, and to perform transfer learning and generalization training according to actual scenarios to realize cross-scenario application.

[0112] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A soft measurement method for liquid bacterial biomass based on machine vision, characterized in that: The steps include: S1: Analyze the dominant variables of biomass parameters of liquid strains during the whole fermentation cycle and their evolution law. The dominant variables of biomass parameters Bio-Indices include: strain concentration %, strain fresh weight g / L and strain dry weight g / L; S2: Offline sampling and manual measurement of auxiliary variable features M-Features of the dominant variables of the associated biomass parameters, including: fermentation broth volume V ls , strain volume V sp , strain quality M sp , and fermentation liquid quality M ls , establish the soft-sensing model M1 of the dominant variable Bio-Indices of biomass parameters; S3: Collect visible-near infrared I region-near infrared II region atlas data of liquid bacteria at different fermentation stages, and establish a multi-source atlas data set to characterize the dynamic evolution of liquid bacteria biomass; S4: Build a multi-task machine vision model V1 based on a multi-source graph dataset, and extract key graph features to obtain auxiliary variable features E-Features based on machine vision, including: fermentation window area S ls , strain area S sp , Bacteria Spectral Index I sp and fermentation broth spectral index I ls . S5: Combining the biomass parameter soft measurement model M1 and the multi-task machine vision model V1, a cascade machine vision soft measurement model MV is obtained to perform soft measurement analysis of Bio-Indices; S6: deploy the cascaded machine vision soft measurement model MV on the terminal embedded intelligent device; S7: Design interactive software for machine vision task execution, realize online real-time reasoning of cascaded machine vision soft sensor model MV, cross-scenario transfer learning and cross-scenario application.

2. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1 is characterized in that: The whole fermentation cycle in S1 includes: adaptation period, exponential growth period, maturity period, and senescence period; the evolution law of the biomass parameter leading variable Bio-Indices refers to the strain concentration C on , fresh weight of strains BM f BM d The dynamic changes that occur as fermentation time progresses during the entire fermentation cycle.

3. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1 is characterized in that: The S2 is specifically: the fermentation liquid volume V in the auxiliary variable feature M-Features ls , refers to the volume of the fermentation broth of the sample; the volume of the strain V in the auxiliary variable feature M-Features sp , refers to the total volume of all bacterial clusters in a certain volume of sample; the bacterial mass M in the auxiliary variable feature M-Features sp , refers to the total mass of all bacterial clusters in a certain volume of sample. The fermentation liquid mass M in the auxiliary variable feature M-Features ls , refers to the quality of the fermentation broth of the sample; The soft measurement model M1 is an indirect measurement method, which realizes the inference of the biomass parameter dominant variable Bio-Indices by establishing a mathematical equation between auxiliary variables and dominant variables; the mathematical equation establishment process is: the bacterial species concentration C in Bio-Indices on Depends on the volume V of the bacteria in M-Features sp The fermentation liquid volume V ls ; Fresh weight of strains in Bio-Indices BM f Depends on the quality of the strain M in M-Features sp ; Dry weight of strains in Bio-Indices BM d Depends on the fresh weight of the strain BM f and moisture content of bacterial mass, the simultaneous equations are as follows: Where a is the volume of the culture V sp Correlation coefficient between offline analysis and online measurement; b is the volume of fermentation liquid V ls Correlation coefficient between offline analysis and online measurement; c is the mass of the strain M sp Correlation coefficient between offline analysis and online measurement; d is the moisture content coefficient of the bacterial mass.

4. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1 is characterized in that: The S3 is specifically: the multi-source atlas data set includes: visible light two-dimensional image, visible light-near infrared I region-near infrared II region hyperspectral one-dimensional spectrum data; The visible light band range is 380-780nm, the near infrared I band range is 780-1000nm, the near infrared II band range is 1000-1700nm, and the band resolution is 1nm; the selection of the visible light band can characterize the intuitive change characteristics of the color of the bacteria and the color of the fermentation liquid during the fermentation process; the selection of the transmission band of the near infrared I zone-near infrared II zone can characterize the evolution characteristics of the body compactness and surface burrs of the bacterial mass during the fermentation process; the selection of the diffuse reflection band of the visible-near infrared I zone-near infrared II zone can characterize the evolution characteristics of the colloidal dispersion system of the fermentation liquid, the fermentation product and the fermentation substrate; the evolution characteristics of the colloidal dispersion system include the evolution of the fermentation substrate and the fermentation product of the fermentation liquid from a heterogeneous suspension, a turbid liquid, a liquid sol to a homogeneous true solution as the fermentation progresses, thereby obtaining the corresponding spectral index characteristics.

5. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1 is characterized in that: The construction of the multi-task machine vision model V1 in S4 is to realize the extraction and analysis of auxiliary variable features by cascading multi-task machine vision models; The multi-task machine vision model performs a series of visual tasks, including classification tasks, target detection tasks, semantic segmentation tasks and morphological image processing tasks; The classification task realizes the task of classifying the liquid strain fermentation tank and other equipment; The target detection task realizes the task of identifying and locating the fermentation window position of the fermentation tank; The semantic segmentation task realizes the task of segmenting the liquid strain area in the fermentation window; The morphological image processing task realizes the separation of bacterial strains and fermentation broth, extracts key atlas features, and obtains auxiliary variable features E-Features based on machine vision.

6. The method for soft measurement of liquid bacterial biomass based on machine vision according to claim 5, characterized in that: The key atlas features include: LBP features, HOG features, morphological factor features and spectral index features; The fermentation window area S in the auxiliary variable feature E-Features ls , refers to the area of ​​the observed fermentation window; the strain area S in the auxiliary variable feature E-Features sp , refers to the total area of ​​the bacterial cluster observed in the observed fermentation window; the bacterial species spectral index I in the auxiliary variable feature E-Features sp , refers to the spectral index of the visible-near infrared I region-near infrared II region of the bacterial mass in the observed fermentation window; the fermentation liquid spectral index I in the auxiliary variable feature E-Features ls It refers to the spectral index outside the visible-near infrared I region and near infrared II region of the fermentation liquid in the observed fermentation window.

7. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The process of cascading the machine vision soft measurement model MV in S5 is specifically to associate the auxiliary variable features E-Features obtained by the multi-task machine vision model V1 with the auxiliary variable features M-Features measured manually, establish the reasoning equation group of E-Features and M-Features, and input the biomass parameter soft measurement model M1 to derive the biomass parameters; The process of establishing association between E-Features and M-Features is as follows: the volume of bacteria in M-Features V sp The area S of the bacteria in the fermentation window observed in E-Features sp Mathematical equations can be established; the strain quality M in M-Features sp The area of ​​bacteria in E-Features sp and bacterial species spectral index I sp Mathematical equations can be established; the fermentation liquid quality M in M-Features ls and fermentation broth spectral index I ls And fermentation window area S ls Mathematical equations can be established; the mathematical equations between the above E-Features and M-Features are combined as follows: Where e is the bacterial area S measured online sp The corresponding bacterial volume V sp A constant correction factor between ; f(I sp ) is the correlation function between the online measurement of bacterial species spectral index and bacterial species density; f(I ls ) is the correlation function between the online measurement of the fermentation liquid spectral index and the fermentation liquid density; g is the fermentation window area S ls The corresponding fermentation liquid volume V ls A constant correction factor between ; By bringing the M-Features derived from E-Features into the soft measurement model M1 for derivation, it is possible to achieve indirect measurement of biomass parameters through the cascade machine vision soft measurement model, avoiding the lag and contamination risk of offline sampling.

8. The method for soft measurement of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The terminal embedded intelligent device in the S6 includes: an edge computing device with an independent graphics card, a built-in wireless network card, an SD card slot for data expansion, and an anti-seismic, lightning-proof and anti-static design for industrial-grade scenario applications; the model deployment process includes: adopting an unstructured pruning strategy to perform lightweight improvements on the machine vision model.

9. The method for soft measurement of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The interactive software for task execution of the cascaded machine vision soft-sensing model in S7 is a human-machine interface designed through pyQT6 to realize the functions of interactive execution of machine vision tasks and cross-scenario transfer learning; The functions of the interactive software for machine vision task execution include: realizing database expansion, model mobilization, model execution, model convergence training, model preservation, model reasoning and biomass parameter measurement effect display on the human-computer interaction interface; The cross-scenario transfer learning of the cascade machine vision soft sensor model MV refers to the generalization migration of the model through real-time interactive establishment of data sets, interactive model loading, and convergence training under cross-scenario conditions; the cross-scenario application refers to the inference of target biomass parameters under different strains and different field environments. If the inferred biomass parameter dominant variable does not meet the accuracy requirement, the data set is continued to be expanded, and the model loading and convergence training are repeated until the accuracy requirement is met; if the inferred biomass parameter has met the accuracy requirement, it means that the model meets the online real-time non-invasive monitoring of the biomass parameters of the current scene.

Images