A soft measurement method for liquid strain biomass based on machine vision

Abstract

The application provides a liquid strain biomass soft measurement method based on machine vision, comprising the following steps: offline sampling and analyzing liquid strain biomass parameters and evolution rules in the whole fermentation period; establishing a biomass parameter soft measurement model M1 by means of manually measured auxiliary variable characteristics M-Features; collecting visible light images and visible-near infrared I-near infrared II multi-source spectrum data of liquid strains in different fermentation stages, constructing an atlas data set, and building a multi-task machine vision model V1 for extracting auxiliary variable characteristics E-Features based on machine vision; combining the biomass parameter soft measurement model M1 and the multi-task machine vision model V1, establishing the correlation between E-Features and M-Features, and obtaining a cascade machine vision soft measurement model MV; and deploying the MV model on a terminal embedded intelligent device and designing interactive software for task execution. The method can evaluate the trend of the evolution of the biomass parameter of the liquid strain fermentation process in real time, so as to guide the production application of the strain.

Description

Technical Field

[0001] The present invention relates to the technical field of soft measurement of biomass parameters in fermentation engineering, and in particular to a method for detecting biomass parameters in a liquid strain fermentation process using cascade machine vision. Background Art

[0002] The quality and quantity of liquid culture determine the quality and yield of edible fungi. To obtain target cultures with high biomass, purity, and vigor in the shortest possible time, in-situ, real-time, non-contact monitoring of the dynamic evolution of biomass during the fermentation process is essential for achieving stable, high-quality output. The fermentation process of liquid culture is typically a dynamic growth and evolution process, involving fission growth of the culture, nutrient consumption in the fermentation broth, and accumulation of fermentation products. Therefore, the fermentation process is a complex, nonlinear system with large hysteresis, multiple variables, and strong coupling, and its internal reaction mechanisms are very complex. In actual fermentation production processes, the parameters that can be automatically monitored are mainly physicochemical parameters that can be directly and easily measured by existing sensors, such as temperature, pH, dissolved oxygen, and tank pressure. However, other important biochemical parameters that affect fermentation quality, such as substrate concentration, cell concentration, and product concentration, must be obtained using offline analysis methods.

[0003] Offline analysis has a large hysteresis and is prone to fermentation contamination, thereby affecting fermentation efficiency and product quality. Soft measurement technology is one of the effective ways to solve the above problems. Soft measurement technology is an indirect measurement method that uses easily measured parameters (auxiliary variables) to achieve inferred measurement of key parameters (dominant variables) that are not easy to measure through modeling. Its core lies in establishing a soft measurement model that characterizes the mathematical relationship between auxiliary variables and dominant variables. Therefore, it is generally necessary to solve the mathematical modeling problem of the soft measurement model in order to achieve a robust characterization of the dominant variable. Currently, there are three main soft measurement modeling methods: one is soft measurement based on mechanism analysis (white box model), the second is soft measurement based on neural network or deep learning (black box model), and the third is soft measurement based on a hybrid model of mechanism analysis and machine learning (gray box model).

[0004] The establishment of white-box, black-box, and gray-box soft-sensor models all rely on the acquisition of one-dimensional sequences of physical and chemical parameters, such as easily measured physical and chemical parameters like temperature, pH, dissolved oxygen, and tank pressure, which are then used for mechanistic modeling or big data analysis. Because these one-dimensional sequences of physical and chemical parameters are external environmental parameters of the fermentation process or process parameters that indirectly reflect biological parameters, they cannot robustly represent the biological parameters of the fermentation process. Numerous studies have demonstrated that the use of visible-infrared spectral data can achieve online, real-time monitoring of biological parameters. However, online monitoring of the biomass of liquid edible mushroom cultures presents several challenges. These include: the constant pneumatic agitation of the liquid culture affects the stable acquisition of mycelial spectral signatures; the strong absorption of water by near-infrared wavelengths affects the robustness and accuracy of biological parameter measurements; and the need for online, real-time monitoring of fermentation biomass parameters conflicts with the accuracy and lightweight nature of model edge deployment. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method for detecting the biomass of liquid strains based on machine vision, a soft measurement scheme for inferring parameters that are difficult to measure by a cascaded machine vision model that integrates multi-source image information, namely a "color box" model, to make up for the limitations of the "white box" based on mechanism analysis, the "black box" based on neural network or big data analysis, and the "gray box" scheme based on a hybrid model of the above two, so as to realize online real-time visual monitoring of the biomass parameters of the liquid strain fermentation process, so as to guide the production and application of liquid strains in the fermentation process.

[0006] To achieve the above object, the technical solution of the present invention is: a soft measurement method of liquid bacterial biomass based on machine vision comprises the following steps:

[0007] S1: Analyze the dominant variables of liquid culture biomass parameters and their evolution patterns during the entire fermentation cycle. The dominant variables of biomass parameters, Bio-Indices, include: culture concentration%, culture fresh weight g / L, and culture dry weight g / L.

[0008] S2: Offline sampling and manual measurement of auxiliary variable features M-Features of associated liquid strain biomass parameters, including: fermentation liquid volume V ls , strain volume V sp , strain quality M sp and fermentation broth quality M ls , establish a soft sensor model M1 of the dominant variable of biomass parameters based on auxiliary variable features M-Features;

[0009] S3: Collect hyperspectral data of visible-near infrared I and near infrared II regions of liquid bacteria at different fermentation stages to construct a multi-source atlas dataset characterizing the dynamic evolution of liquid bacteria biomass;

[0010] S4: Build a multi-task machine vision model V1 based on a multi-source atlas dataset. By extracting key atlas features, we obtain auxiliary variable features E-Features based on machine vision, including: fermentation window area S ls , strain area S sp , bacterial species spectral index I sp and fermentation broth spectral index I ls .

[0011] S5: Combining the biomass parameter soft measurement model M1 and the multi-task machine vision model V1, a cascade machine vision soft measurement model MV is obtained to perform soft measurement analysis of Bio-Indices;

[0012] S6: deploy the cascaded machine vision soft measurement model MV on the terminal embedded intelligent device;

[0013] S7: Design interactive software for machine vision task execution to realize cross-scene transfer learning and online real-time reasoning of cascaded machine vision soft measurement model MV, cross-scene transfer learning and cross-scene application.

[0014] Preferably, the S1 is specifically:

[0015] Offline sampling and analysis of liquid bacterial strains specifically include: aseptic sampling, filtration, pure water washing, centrifugation, freeze drying, laboratory analysis and measurement of bacterial strain biomass parameters; the full fermentation cycle includes: adaptation period (1-2 days), rapid growth period (3-6 days), maturity period (7-8 days), aging period (8-10 days); the evolution law of the biomass parameter leading variable Bio-Indices refers to the bacterial strain concentration C on (%), fresh weight of strain BM f (g / L) and strain dry weight BM d (g / L) The dynamic changes of the fermentation content as the fermentation time progresses during the entire fermentation cycle.

[0016] Preferably, the S2 is specifically:

[0017] Fermentation broth volume V in auxiliary variable feature M-Features ls , is the volume of the fermentation liquid of the sample, the sampling volume can be 400mL; the strain volume V in the auxiliary variable feature M-Features sp , is the total volume of all bacterial clusters in a certain volume of sample; the bacterial mass M in the auxiliary variable feature M-Features spM is the total mass of all the microbial aggregates in a certain volume of the sample, and M is the mass of the fermentation broth in the M-Features ls M is the mass of the fermentation broth in the sample;

[0018] The soft measurement model M1 is established to realize the estimation of the Bio-Indices by establishing a mathematical equation between the auxiliary variables and the leading variables; the establishment of the mathematical equation depends on the mechanism analysis and mathematical derivation between the auxiliary variable features and the leading variable features, and the specific process is as follows: the concentration of the microbial species C on depends on the volume of the microbial species V sp in the M-Features; ls ; the fresh weight of the microbial species BM f in the Bio-Indices depends on the mass of the microbial species M sp in the M-Features; the dry weight of the microbial species BM d in the Bio-Indices depends on the fresh weight of the microbial species BM f and the water content of the microbial aggregates, and the equations are as follows:

[0019]

[0020] In the formula,

[0021] a is the volume of the microbial species V sp , the correlation coefficient between the offline analysis and the online measurement, the value range is (1.0-1.2);

[0022] b is the volume of the fermentation broth V ls , the correlation coefficient between the offline analysis and the online measurement, the pressure change has no obvious effect on it, and the value is generally 1.0;

[0023] c is the mass of the microbial species M sp , the correlation coefficient between the offline analysis and the online measurement, the pressure change has no obvious effect on it, and the value is generally 1.0;

[0024] d is the water content coefficient of the microbial aggregates, which is linearly adjusted according to the liquid microbial species and the fermentation period, and the value range is (8.0-9.5).

[0025] Preferably, the S3 is specifically as follows:

[0026] A multi-source atlas dataset is established, including visible light RGB two-dimensional images and one-dimensional spectral data of visible light-near infrared region I-near infrared region II bands; the visible light band range is 380-780nm, the near infrared region I band range is 780-1000nm, the near infrared region II band range is 1000-1700nm, and the band resolution is 1nm; wherein, the selection of the visible light band can characterize the color change characteristics of the strain and the fermentation liquid color, the selection of the near infrared region I and region II bands for transmission can characterize the body compactness of the bacterial mass and the evolution characteristics of the surface burrs during the fermentation process, and the selection of the near infrared region I and region II bands for diffuse reflection can characterize the evolution of the fermentation products and fermentation substrates of the fermentation liquid from a heterogeneous suspension, a turbid liquid, a liquid sol, to a homogeneous true solution as the fermentation progresses, thereby obtaining corresponding spectral index characteristics.

[0027] Preferably, the S4 is specifically:

[0028] The multi-task machine vision model V1 is constructed by cascading multi-task machine vision models to realize the extraction and analysis of auxiliary variable features; the multi-task machine vision model performs a series of visual tasks respectively, including classification tasks, target detection tasks, semantic segmentation tasks and morphological image processing tasks; the classification task realizes the task of classifying liquid culture fermentation tanks and other equipment; the target detection task realizes the task of identifying and locating the fermentation window position of the fermentation tank; the semantic segmentation task realizes the task of segmenting the liquid culture area in the fermentation window; the morphological image processing task realizes the separation of culture and fermentation liquid, and extracts key spectral features, including LBP (local binary pattern) features, HOG (histogram of oriented gradients) features, roundness factor, morphological factor features, grayscale, and spectral index features; thereby deriving auxiliary variable features E-Features based on machine vision, including culture area, fermentation window area, culture and fermentation liquid spectral index.

[0029] Preferably, the S5 is specifically:

[0030] The auxiliary variable features E-Features obtained by the multi-task machine vision model V1 are associated with the auxiliary variable features M-Features measured manually, specifically: the volume of the bacterial strain V in M-Features sp The fermentation window area S observed in E-Features ls Inner bacterial area S sp Mathematical equations can be established; the volume of fermentation liquid V in M-Features ls The volume of the fermentation broth sample; M is the mass of the strain in M-Features sp and the bacterial species area S in E-Features sp and bacterial species spectral index Isp Mathematical equations can be established; M-Features of the fermentation broth quality M ls and the fermentation broth spectrum index I ls and the fermentation window area S ls Mathematical equations can be established; the mathematical equations between the above E-Features and M-Features are as follows:

[0031]

[0032] In the formula, e is the constant correction coefficient between the online measured strain area S sp and the corresponding strain volume V sp ;

[0033] f(I sp ) is the correlation function of the online measured strain spectrum index and the strain density;

[0034] f(I ls ) is the correlation function of the online measured fermentation broth spectrum index and the fermentation broth density;

[0035] g is the constant correction coefficient between the fermentation window area S ls and the corresponding fermentation broth volume V ls ;

[0036] The M-Features derived from the E-Features are input into the biomass parameter soft measurement model M1 to derive the biomass parameter Bio-Indices; thereby realizing the indirect measurement of the biomass parameter leading variable through the machine vision extraction of the auxiliary variable characteristics.

[0037] Preferably, the terminal embedded intelligent device in S6 comprises an edge computing device with an independent graphics card, a built-in wireless network card, a data expansion SD card slot, an anti-vibration, lightning and static electricity design for industrial scene application, and a model deployment process comprising a non-structured pruning strategy for lightweight improvement of the machine vision model.

[0038] Preferably, the interactive software for executing the machine vision model task in S7 is a human-computer interface designed through pyQT6 to realize the function of interactive execution of machine vision tasks; the functions of the interactive software for executing machine vision tasks include: on the human-computer interaction interface, realizing database expansion, model mobilization, model execution, model convergence training, model preservation and model reasoning and biomass parameter measurement effect display; the cross-scene transfer learning of the cascaded machine vision soft measurement model MV refers to the generalization migration of the model through real-time interactive establishment of data sets, interactive model loading, convergence training under cross-scene conditions; the cross-scene application refers to the target biomass parameter estimation under different strains and different field environments. If the estimated biomass parameter dominant variable does not meet the accuracy requirements, the data set will continue to be expanded, and the model loading and convergence training will be repeated until the accuracy requirements are met; if the estimated biomass parameter has met the accuracy requirements, it means that the model meets the online real-time non-invasive monitoring of the biomass parameter of the current scene.

[0039] Compared with the prior art, the present invention has the following beneficial effects: it provides a soft measurement method for liquid strain biomass parameters based on cascade machine vision, so as to realize online non-invasive monitoring of biological parameters that are difficult to measure directly. This is achieved by fusing a "color box model" of multi-source maps and machine learning, namely a cascade machine vision soft measurement model, to realize the inference of parameters that are difficult to measure. Auxiliary variable features E-Features are obtained through machine vision, associated with M-Features, and then input into the soft measurement model for indirect measurement of the biomass parameter leading variable Bio-Indices. Since the "color box" soft measurement method has both the mechanism analysis of two-dimensional maps and the feature extraction capabilities of neural networks, it can make the inference effect more interpretable and robust than the feature extraction capabilities usually based only on neural network learning, or the soft measurement model based only on one-dimensional environmental parameters, thereby providing an ideal solution for non-invasive real-time monitoring of biomass parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Attachment Figure 1 This is a flow chart of offline sampling and analysis of biomass parameters according to a preferred embodiment of the present invention; wherein, step a represents aseptic sampling, step b represents molecular sieve filtration, step c represents the strain after filtration and pure water cleaning, step d represents volume measurement of the fermentation broth after filtration, step e represents offline analysis and measurement of the fresh weight (g / L) of the strain to obtain the biomass parameter, step f represents offline analysis and measurement of the strain concentration (%) of the biomass parameter, step g represents freeze-drying the strain obtained in step c, and step h represents offline analysis and measurement of the dry weight (g / L) of the strain to obtain the biomass parameter;

[0041] Attachment Figure 2Machine vision platform connection and component diagram of preferred embodiment of the present application; wherein, (a) WIFI antenna component, (b) 4020 fan, (c) heat sink, (d) Jetson Nano module, (e) Jetson Nano carrier board, (f) 128G microSD card, (g) portable monitoring screen, (h) 2 degrees of freedom holder, (i) CSI camera;

[0042] Appendix Figure 3 Experimental results of classification task in multi-task machine vision of the present application; wherein (a) represents the loss and evaluation index of the classification task training process, (b) represents the confusion matrix of the training result, (c) represents the classification task training effect;

[0043] Appendix Figure 4 Experimental results of target detection task in multi-task machine vision of the present application; wherein (a) represents the loss and evaluation index of the target detection task training process, (b) represents the confusion matrix of the training result, (c) represents the label and bounding box correlation diagram of the target detection task training process, (d) represents the training result of the target detection task;

[0044] Appendix Figure 5 Experimental results of semantic segmentation task in multi-task machine vision of the present application; wherein (a) represents the loss and evaluation index of the semantic segmentation task training process, (b) represents the confusion matrix of the training result, (c) represents the label and bounding box correlation diagram of the semantic segmentation task training process, (d) represents the training result of the target detection task;

[0045] Appendix Figure 6 Experimental effect diagram of machine vision task of the present application; wherein, step a represents the fermentation tank window area, step b represents the liquid strain target area segmentation, and step c represents the liquid strain target area of interest extraction;

[0046] Appendix Figure 7 Experimental results of preferred biomass parameter estimation based on machine vision of the present application; wherein, a represents the machine vision processing process of the fermentation tank in the whole fermentation period, b represents the biomass change dynamic curve obtained based on machine vision processing, c represents the correlation coefficient obtained based on machine vision extraction of the biomass parameter and based on artificial measurement of the biomass parameter, and d represents the biomass parameter change dynamic curve corrected by the correlation coefficient;

[0047] Appendix Figure 8 Interactive software interface for executing independent machine vision tasks of the present application; wherein, (a) represents the classification task interface, (b) represents the target detection task interface, and (c) represents the semantic segmentation task interface;

[0048] Appendix Figure 9An interactive software interface for performing machine vision synthesis tasks for the present invention;

[0049] Attachment Figure 10 Comparison of the biomass parameters inferred by machine vision and those based on manual measurement in the present invention; where a is the manually measured biomass parameter, b is the biomass parameter inferred by machine vision, c and d are the normalized results of the biomass parameters, e and f are the Boltzmann fitting results of the strain biomass growth curve, and g and h represent the corresponding relationship between the biomass parameters inferred by machine vision and those based on manual measurement, as well as their Pearson correlation.

[0050] Figure 11 This is a diagram illustrating the process of biomass parameter estimation based on machine vision in the present invention, wherein step a represents the establishment of a data set, step b represents the establishment of a machine vision model, step c represents the construction of a machine vision platform, step d represents the results of cascaded machine vision execution, step e represents the establishment of inference association equations for various biomass parameters, and step f represents the interactive interface for executing biomass parameter estimation;

[0051] Figure 12 This is a flow chart of the cascaded machine vision model MV of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0055] refer to Figures 1 to 12 The soft measurement of biomass parameters in the fermentation process of velvet antler mushroom liquid spawn based on machine vision includes the following steps:

[0056] S1: Offline sampling and analysis of the evolution of the leading variables of the biomass parameters of the liquid strains during the whole fermentation cycle; measuring the biomass indicators of the liquid strains of the velvet mushroom at different stages during the whole fermentation cycle, wherein the whole fermentation cycle includes the adaptation period (2 days), the rapid growth period (3 days), the maturity period (2 days), and the senescence period (2 days); further, the offline sampling and analysis includes: aseptic sampling, alcohol disinfection, flame isolation, and sampling; constant temperature centrifugation: 1500r / min, 15min; filtration, 150 mesh molecular sieve; washing, RO pure water washing, three times; freeze drying, -80 refrigerator freezing hard, -50℃ freeze drying to room temperature; measuring the leading variable indicators of the biomass parameters, recording the experimental data and archiving them; the offline sampling and analysis process is as follows Figure 1 As shown;

[0057] Furthermore, the biomass parameter leading variable indicators of the offline sampling analysis include the strain concentration C on (%), fresh weight of strain BM f (g / L) and strain dry weight BM d (g / L); further, the biomass parameter calculation equation is:

[0058]

[0059]

[0060] Where V ls Indicates the volume of fermentation broth in the sample taken;

[0061] V sp Table of the volume of bacteria in the sampled samples;

[0062] Indicates the mass of water reduced by freeze drying.

[0063] S2: Offline sampling and manual measurement of auxiliary variable features M-Features associated with liquid strain biomass parameters, including the liquid strain fermentation liquid sampling volume V ls , bacterial volume V sp , strain quality M sp and fermentation broth quality M ls Parameters; establish the soft measurement model M1 of the biomass parameter dominant variable Bio-Indices;

[0064] Auxiliary variable features M-Features V ls Refers to the volume of fermentation broth of the sampled sample. The sampling volume of each sample can be divided into 400 mL. The auxiliary variable feature M-Features V spis the total volume of all bacterial groups in a certain volume of sampling sample; M sp is the total mass of all bacterial groups in a certain volume of sampling sample; M ls is the mass of the fermentation broth after removing the mass of the bacterial strain in a certain volume of sampling sample;

[0065] The soft measurement model M1 is an indirect measurement method, which realizes the estimation of the biomass parameter leading variable Bio-Indices by establishing a mathematical equation of auxiliary variables and leading variables; the M1 establishment process can be obtained by mechanism analysis and mathematical derivation between auxiliary variable features and leading variable features;

[0066] Further, the mechanism analysis specifically refers to that the fresh weight of the bacterial strain BM f depends on the mass of the bacterial strain M sp , and is corrected by a constant coefficient; the concentration of the bacterial strain C on depends on the volume of the bacterial strain V sp and the volume of the fermentation broth V ls , and can be inferred by M-Features three-dimensional volume or E-features two-dimensional area; and the dry weight of the bacterial strain BM d depends on the mass of the bacterial strain M sp and the moisture content of the bacterial group, and the moisture content of the bacterial group can be obtained by statistical analysis;

[0067] Further, the specific process of the mathematical derivation is that the concentration of the bacterial strain C on in the Bio-Indices depends on the volume of the bacterial strain V sp in the M-Features and the volume of the fermentation broth V ls ; the fresh weight of the bacterial strain BM f in the Bio-Indices depends on the mass of the bacterial strain M sp in the M-Features; the dry weight of the bacterial strain BM d in the Bio-Indices depends on the fresh weight of the bacterial strain BM f and the moisture content of the bacterial group, and the simultaneous equations are as follows:

[0068]

[0069] In the formula,

[0070] a is the correlation coefficient between offline analysis and online measurement of the volume of the bacterial strain V sp , and the value interval is (1.0-1.2);

[0071] b is the correlation coefficient between offline analysis and online measurement of the volume of the fermentation broth V lsThe correlation coefficient between the offline analysis and the online measurement does not take into account the pressure change and is generally taken as 1.0;

[0072] c is the mass of the strain M sp The correlation coefficient between the offline analysis and the online measurement does not take into account the pressure change and is generally taken as 1.0;

[0073] d is the moisture content coefficient of the bacterial mass, which is linearly adjusted according to the liquid culture type and fermentation cycle, and the value range is (8.0-9.5).

[0074] S3: Collect visible light images and near-infrared multispectral reflectance images of liquid bacteria at different fermentation stages to construct a multi-source atlas dataset characterizing the dynamic evolution of liquid bacteria biomass. Use a hyperspectral imager covering the visible-near-infrared I-near-infrared II band (380-1700nm) to collect spectral information of biomass indicators at different fermentation stages and establish a multi-source atlas dataset.

[0075] Furthermore, the multi-source atlas dataset includes visible light RGB two-dimensional images and visible light-near infrared I region-near infrared II region hyperspectral one-dimensional spectrum data; the visible light band range is 380-780nm, the near infrared I region band range is 780-1000nm, the near infrared II region band range is 1000-1700nm, and the band resolution is 1nm; wherein, the selection of the visible light band can characterize the color change characteristics of the strain and the fermentation liquid color, and the selection of 580-595nm in the visible light band can characterize the color change characteristics of the strain and the fermentation liquid color;

[0076] Furthermore, the selection of the transmission band of the near-infrared zone I and zone II hyperspectral spectrum can characterize the evolution characteristics of the body compactness of the bacterial mass and the burrs on the machine surface during the fermentation process; the selection of the diffuse reflection band of the near-infrared zone I and zone II hyperspectral spectrum can characterize the evolution characteristics of the colloidal dispersion system of the fermentation liquid, fermentation products and fermentation substrates, such as the evolution of the fermentation products and fermentation substrates of the fermentation liquid from a heterogeneous suspension, suspensoid, liquid sol to a homogeneous true solution as the fermentation progresses, thereby obtaining the corresponding spectral index characteristics.

[0077] S4: Construct a multi-task machine vision model V1 based on a multi-source atlas dataset to extract auxiliary variable features E-Features. Construct a multi-task machine vision model V1 based on deep learning, which is a machine vision model that cascades multiple tasks to achieve the extraction and analysis of key atlas features. The selection of the multi-task machine vision model is to enable the machine vision to be positioned in the key window area for analysis, reduce the amount of calculation and improve the accuracy. The specific process is to first use the multi-task machine vision model to enable the vision model to identify and locate the window position, that is, find the area of ​​interest, and then perform key target detection and analysis in the area of ​​interest; the segmentation task of the cascade task is to detect target objects such as target bacteria and fermentation liquid in the fermenter based on the atlas data obtained by the target detection task in the previous stage. This example uses threshold segmentation and morphological segmentation for processing to obtain image information of key targets.

[0078] Furthermore, the construction of a multi-task machine vision model is based on the YOLOv8 model, and implements multi-task machine vision tasks to achieve the extraction and analysis of key atlas information;

[0079] Furthermore, the multi-task machine vision model performs a series of visual tasks, including classification tasks, object detection tasks, semantic segmentation tasks and morphological image processing tasks;

[0080] Furthermore, the classification task is to realize the classification of liquid bacterial fermentation tanks and other equipment; the training set reasoning results of the model classification task are as follows: Figure 3 ;in Figure 3 (a) represents the loss and evaluation index of the classification task training process, Figure 3 (b) represents the confusion matrix of the training results, Figure 3 (c) shows the training effect of classification task;

[0081] Furthermore, the target detection task is to realize the task of identifying and locating the fermentation window position of the fermentation tank; the training set reasoning result of the target detection task is as follows: Figure 4 ;in Figure 4 (a) represents the loss and evaluation index of the target detection task training process, Figure 4 (b) represents the confusion matrix of the training results, Figure 4 (c) Correlation diagram of labels and bounding boxes during the training process of the target detection task. Figure 4 (d) represents the training results of the target detection task;

[0082] Furthermore, the semantic segmentation task is to segment the liquid culture area within the fermentation window; the training set reasoning result of the semantic segmentation task is as follows: Figure 5 ;in Figure 5(a) represents the loss and evaluation indicators of the semantic segmentation task training process, Figure 5 (b) represents the confusion matrix of the training results, Figure 5 (c) Correlation diagram of labels and bounding boxes during the semantic segmentation task training process. Figure 5 (d) represents the training results of the target detection task;

[0083] Furthermore, the morphological image processing task is to separate the strains from the fermentation broth and to extract and analyze key image features; the process and results of morphological image processing are as follows: Figure 6 ;in, Figure 6 Step a represents the fermentation tank window area, Figure 6 Step b represents the segmentation of the target area of ​​the liquid bacteria. Figure 6 Step c represents the extraction of the target region of interest from the liquid bacterial strain;

[0084] Furthermore, the threshold segmentation method described above adopts an adaptive threshold segmentation method, which is a type of local threshold segmentation. Unlike global thresholding, which uses a single threshold for the entire matrix, each position of the input matrix has a corresponding threshold. Assuming that the input image is I, the steps of the adaptive threshold segmentation algorithm are as follows:

[0085] Thresh=(1-ratio)*f smooth (I)

[0086]

[0087] Among them, Thresh represents the adaptive threshold matrix, f smooth (I) represents the result of smoothing the image, ratio is the operation ratio, O(r,c) represents the pixel value of a point in the image after processing, where r represents the row vector of the image and c represents the column vector of the image. This formula shows that if the pixel value I(r,c) of a point in the image is greater than the threshold Thresh(r,c) at that point, then the pixel value of that point is 255, otherwise it is 0;

[0088] Furthermore, the morphological image processing task uses a closing operation to address the noise and frame adhesion problems present in the threshold segmentation image through processing methods such as corrosion and dilation. The corrosion operation can eliminate the boundary of the target object, resulting in an area one pixel smaller than the original target, which can separate originally adhered graphics but reduce the size of the original graphics. The dilation operation has the opposite effect and can merge two adhered graphics. For multispectral images of liquid bacteria, circular structural elements are selected for closing operations, which can fill small holes and cracks in the image without changing the overall position and shape of the graphics, thereby improving the detection rate of bacterial clusters and obtaining more accurate key atlas features.

[0089] Furthermore, the key atlas features include: LBP features, HOG features, morphological parameter features (morphological factor, roundness factor, area perimeter factor), spectral index features (grayscale, average grayscale), thereby deriving auxiliary variable features E-Features based on machine vision, including strain area, fermentation window area, strain and fermentation liquid spectral index;

[0090] Furthermore, the derivation process between the key spectral features and E-Features is a polynomial response surface analysis-polynomial regression analysis; the response surface analysis method is used to analyze the changes in the target variable E-Features when the key spectral features change consistently or inconsistently, and the E-Features regression equation can be established by weight distribution combined with multivariate statistical analysis;

[0091] Furthermore, the polynomial regression analysis is specifically implemented by least squares fitting to find a suitable approximate function, using the significance test of regression analysis to understand the strength of the relationship between the independent variable and the response variable, and to test whether the fitting model is appropriate; first, a low-order polynomial approximation of the independent variable in some range is used, that is, a first-order regression model. When the experimental area is close to the optimal response value, the curvature of the true response surface increases, and a second-order model is considered; second, if the second-order regression model is well-fitted, the second-order model can be used to obtain the optimal combination point and the characterized response surface; further, if the second-order model still lacks adaptability when fitting, a local optimal operating state can be obtained or a higher regression model, such as a cubic or quartic multi-order model, can be fitted; the first-order equation model adopted in this embodiment is:

[0092]

[0093] Where: is the response target of the first-order model, and the parameter x i (i∈(1,k)) are the independent variables, is the correlation coefficient.

[0094] The second-order equation model is:

[0095]

[0096] Where: is the response target of the second-order model, and the parameter x i (i∈(1,k)), x j (j∈(i,k)) are the independent variables, is the correlation coefficient.

[0097] The third-order equation model is:

[0098]

[0099] Where: is the response target of the three-world model, are the parameters x i The upper and lower bounds of the value range, are the correlation coefficients.

[0100] S5: Combine the biomass parameter soft measurement model M1 and the multi-task machine vision model V1 to establish the association between the auxiliary variable features E-Features obtained by machine vision and the auxiliary variable features M-Features obtained by manual measurement, and obtain the soft measurement model MV for biomass parameter measurement based on cascade machine vision, as shown in Figure 5 As shown;

[0101] Furthermore, the correlation equation between E-Features and M-Features is corrected and correlated by introducing weight coefficients through mechanism analysis and data statistics. The specific process is: the volume of the strain V in M-Features sp The fermentation window area S observed in E-Features ls Inner bacterial area S sp Mathematical equations can be established; the volume of fermentation liquid V in M-Features ls The volume of the fermentation broth sample; M is the mass of the strain in M-Features sp and the bacterial species area S in E-Features sp and bacterial species spectral index I sp Mathematical equations can be established; the fermentation liquid quality M in M-Features ls and fermentation broth spectral index I ls And fermentation window area S ls Mathematical equations can be established; the mathematical equations between the above E-Features and M-Features are as follows:

[0102]

[0103] Where, e is the online measured bacterial area S sp The corresponding bacterial volume V sp A constant correction factor between

[0104] f(I sp ) is the correlation function between the online measurement of bacterial species spectral index and bacterial species density;

[0105] f(I ls) is a correlation function of the online measurement of the fermentation broth spectrum index and the fermentation broth density;

[0106] g is the fermentation window area S ls and the corresponding fermentation broth volume V ls between the constant correction coefficient;

[0107] The M-Features derived from E-Features are input into the biomass parameter soft measurement model M1 to derive the biomass parameter Bio-Indices. Through the training of the cascading machine vision soft measurement model MV, the correlation between the multi-source atlas data set and the auxiliary variable feature E-Features is established, so as to infer the biomass parameter. The experimental and inference results of the liquid strain biomass parameter based on machine vision are as shown in Figure 7 , wherein Figure 7 Step a represents the machine vision processing process of the fermentation tank in the whole fermentation period, Figure 7 Step b represents the biomass change dynamic curve obtained based on the machine vision processing, Figure 7 Step c represents the correlation coefficient obtained based on the biomass parameter extracted based on the machine vision and the biomass parameter measured by the artificial measurement, Figure 7 Step d represents the obtained biomass parameter change dynamic curve after the correlation coefficient correction;

[0108] S6: The machine vision model MV is deployed on an embedded edge computing device. The embedded edge computing device is NIVIDA's JETSON Nano, which includes a CUDA 128-core computing unit, a built-in wireless network card, a data expansion 64G SD card slot, an anti-vibration, lightning and static electricity design IPX4 for industrial scene application. The embedded edge computing device platform effect is as shown in Figure 2 , wherein Figure 2 (a) represents its WIFI antenna part, Figure 2(b) represents its 4020 fan, (c) represents its heat sink, (d) represents its Jetson Nano module, (e) represents its Jetson Nano carrier board, (f) represents its 128G microSD card, (g) represents its portable monitoring screen, (h) represents its 2-degree-of-freedom holder, (i) represents its CSI camera; the lightweight deployment of the model comprises: taking an unstructured pruning strategy to perform lightweight improvement of the machine vision model; research and development of edge computing equipment, including multi-source graph information acquisition control, image preprocessing and deep learning model, and deployment on the intelligent edge computing device of the terminal. The intelligent edge computing embedded device selected in the preferred example is Jetson Nano 4GB of NVDIA, which adopts a four-core 64-bit ARM central processing unit (CPU) and a 128-core integrated NVDIA graphics card (GPU), and can provide a computing performance of 472 GFLOPS. It is equipped with an Ubuntu operating system and a rich CUDA toolkit, and is fully compatible with mainstream open source machine learning frameworks such as Pytorch and TensorFlow, and computer vision processing tool libraries such as OpenCV and ROS, which facilitates the deployment of machine learning inference based on artificial intelligence on the terminal.

[0109] S7: Design the interactive software for performing the machine vision task, realize the transfer learning and online real-time inference of the cascade model, and realize cross-scene transfer learning and cross-scene application. The interactive software for performing the machine vision task is a man-machine interface for performing the machine vision task designed by pyQT6, such as Figure 8 Figure 9 ; Figure 8 (a) represents a classification task interface, Figure 8 (b) represents a target detection task interface, Figure 8 (c) represents a semantic segmentation task interface, and Figure 9 is a comprehensive interface of the interactive software for performing the machine vision task of the application; the model transfer learning is realized by interactively establishing a data set, interactively loading a model and converging training, and evaluating the real-time inference effect of the model. The functions of the interactive software for performing the machine vision task include: realizing database expansion, model mobilization, model execution, model converging training, model saving, model inference and biomass parameter soft measurement effect display on the man-machine interface.

[0110] Further, the offline sampling analysis of the biomass parameters is compared with the online analysis result of the machine vision, and the results are as follows Figure 10 (a) and Figure 10 (b). The biomass parameters have three indexes: concentration, fresh weight and dry weight. Since the value ranges of the three are quite different, the measurement results of the three are normalized, as shown in Figure 10 ​(c). It can be seen that the evolution trend of the three parameters is the same, and the parameter results obtained by machine vision analysis are normalized, which is due to the machine vision processing of the three parameters is associated by a first-order linear relationship, so the normalized results of the three biomass parameters are also basically similar. Statistical analysis between the measurement results of the biomass parameters based on offline analysis and machine vision shows that r 2 is 0.963, RMSE is 0.027, and the calculation result of Pearson correlation coefficient between parameters is 0.91-0.97, which proves that the biomass parameters based on machine vision reasoning have strong correlation with the biomass parameters based on offline sampling analysis.

[0111] In summary, the soft measurement method of liquid strain biomass parameters based on cascade machine vision provided by the present application is a "color box" method by fusing multi-source atlas and machine vision, that is, by establishing the correlation of atlas auxiliary features E-Features obtained by cascade machine vision and manually measured auxiliary features M-Features, inputting the soft measurement model, and finally realizing the indirect measurement of liquid strain fermentation process biomass parameters Bio-Indices. Since the "color box" soft measurement method has the mechanism analysis of atlas and the feature extraction ability of neural network at the same time, it fuses the atlas auxiliary features of cascade machine vision and the manually measured auxiliary features to infer the biomass parameters, and the inference result is more interpretable and robust. Finally, the above machine vision model is deployed on the terminal for visual online monitoring, realizing the non-invasive real-time monitoring of biomass parameters, and according to the actual scene, the generalization training of transfer learning is realized, realizing the cross-scene application.

[0112] The above is the preferred embodiment of the present application, any changes made according to the technical solutions of the present application, as long as the generated function does not exceed the scope of the technical solutions of the present application, belongs to the protection scope of the present application.

Claims

1. A soft measurement method for liquid bacterial biomass based on machine vision, characterized in that: The steps include: S1: Analyze the dominant variables of biomass parameters of liquid bacteria during the entire fermentation cycle and their evolution patterns. The dominant variables of biomass parameters, Bio-Indices, include: bacterial concentration (%), bacterial fresh weight (g / L), and bacterial dry weight (g / L). S2: Offline sampling and manual measurement of auxiliary variable features M-Features associated with the dominant variables of biomass parameters, including: fermentation broth volume V ls , strain volume V sp , strain quality M sp , and fermentation broth quality M ls , establish the soft sensing model M1 of the biomass parameter dominant variable Bio-Indices; S3: Collect visible-near infrared I region and near infrared II region spectral data of liquid bacteria at different fermentation stages to establish a multi-source spectral dataset that characterizes the dynamic evolution of liquid bacteria biomass; S4: Build a multi-task machine vision model V1 based on a multi-source atlas dataset. By extracting key atlas features, we obtain auxiliary variable features E-Features based on machine vision, including: fermentation window area S ls , strain area S sp , bacterial species spectral index I sp and fermentation broth spectral index I ls ; S5: Combine the biomass parameter soft sensing model M1 and the multi-task machine vision model V1 to obtain the cascade machine vision soft sensing model MV to perform soft sensing analysis of Bio-Indices; S6: deploy the cascaded machine vision soft measurement model MV on the terminal embedded intelligent device; S7: Design interactive software for machine vision task execution, implement online real-time reasoning of cascaded machine vision soft sensor models (MVs), and implement cross-scenario transfer learning and cross-scenario applications. The S2 is specifically: the fermentation liquid volume V in the auxiliary variable feature M-Features ls , refers to the volume of the fermentation liquid of the sample; the volume of the strain V in the auxiliary variable feature M-Features sp , refers to the total volume of all bacterial clusters in a certain volume of sample; the bacterial mass M in the auxiliary variable feature M-Features sp , refers to the total mass of all bacterial groups in a certain volume of sample. The fermentation liquid mass M in the auxiliary variable feature M-Features ls , refers to the quality of the fermentation broth of the sample; The soft measurement model M1 is an indirect measurement method, which realizes the inference of the biomass parameter leading variable Bio-Indices by establishing a mathematical equation between auxiliary variables and leading variables; the mathematical equation establishment process is: the bacterial concentration C in Bio-Indices on Depends on the volume V of the bacteria in M-Features sp The volume of fermentation liquid V ls ; Fresh weight of strains in Bio-Indices BM f Depends on the strain quality M in M-Features sp ; Dry weight of bacteria in Bio-Indices BM d Depends on the fresh weight of the strain BM f and moisture content of bacterial mass, the simultaneous equations are as follows: Where a is the volume of the culture V sp Correlation coefficient between offline analysis and online measurement; b is the volume of fermentation liquid V ls Correlation coefficient between offline analysis and online measurement; c is the mass of the strain M sp Correlation coefficient between offline analysis and online measurement; d is the moisture content coefficient of bacterial mass; The process of cascading the machine vision soft measurement model MV in S5 is specifically associating the auxiliary variable features E-Features obtained by the multi-task machine vision model V1 with the auxiliary variable features M-Features measured manually, establishing an inference equation group of E-Features and M-Features, and inputting them into the biomass parameter soft measurement model M1 to derive the biomass parameters; The process of establishing association between E-Features and M-Features is as follows: the volume of bacteria in M-Features V sp The area of ​​bacteria within the fermentation window observed in E-Features S sp Mathematical equations can be established; the strain quality M in M-Features sp and the bacterial species area S in E-Features sp and bacterial species spectral index I sp Mathematical equations can be established; the fermentation liquid quality M in M-Features ls and fermentation broth spectral index I ls And fermentation window area S ls Mathematical equations can be established; the mathematical equations between the above E-Features and M-Features are as follows: Where, e is the online measured bacterial area S sp The corresponding bacterial volume V sp A constant correction factor between f(I sp ) is the correlation function between the online measurement of bacterial species spectral index and bacterial species density; f(I ls ) is the correlation function between the online measurement of the fermentation liquid spectral index and the fermentation liquid density; g is the fermentation window area S ls The corresponding fermentation liquid volume V ls A constant correction factor between By bringing the M-Features derived from E-Features into the soft measurement model M1 for derivation, it is possible to indirectly measure biomass parameters through the cascade machine vision soft measurement model, avoiding the lag and contamination risk of offline sampling.

2. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The full fermentation cycle in S1 includes: adaptation period, exponential growth period, maturity period, and aging period; the evolution law of the biomass parameter leading variable Bio-Indices refers to the strain concentration C on , fresh weight of strains BM f and strain dry weight BM d The dynamic changes that occur as fermentation time progresses during the entire fermentation cycle.

3. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: Said S3 specifically includes: said multi-source atlas data set including: visible light two-dimensional image, visible light-near infrared region I-near infrared region II hyperspectral one-dimensional spectrum data; The visible light band range is 380-780nm, the near-infrared I band range is 780-1000nm, the near-infrared II band range is 1000-1700nm, and the band resolution is 1nm; the selection of the visible light band can characterize the intuitive change characteristics of the color of the bacteria and the color of the fermentation liquid during the fermentation process; the selection of the transmission band of the near-infrared I zone-near-infrared II zone can characterize the body compactness and surface burr evolution characteristics of the bacterial mass during the fermentation process; the selection of the diffuse reflection band of the visible-near-infrared I zone-near-infrared II zone can characterize the evolution characteristics of the colloidal dispersion system of the fermentation liquid, fermentation products and fermentation substrate; the evolution characteristics of the colloidal dispersion system include the evolution of the fermentation substrate and fermentation products of the fermentation liquid from a heterogeneous suspension, a turbid liquid, a liquid sol to a homogeneous true solution as the fermentation progresses, thereby obtaining the corresponding spectral index characteristics.

4. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The construction of the multi-task machine vision model V1 in S4 is to realize the extraction and analysis of auxiliary variable features by cascading multi-task machine vision models; The multi-task machine vision model performs a series of visual tasks, including classification tasks, object detection tasks, semantic segmentation tasks and morphological image processing tasks; The classification task realizes the task of classifying the liquid strain fermentation tank and other equipment; The target detection task realizes the task of identifying and locating the fermentation window position of the fermentation tank; The semantic segmentation task is to segment the liquid culture area within the fermentation window; The morphological image processing task realizes the separation of bacterial strains and fermentation broth, extracts key atlas features, and obtains auxiliary variable features E-Features based on machine vision.

5. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 4, characterized in that: The key atlas features include: LBP features, HOG features, morphological factor features and spectral index features; The fermentation window area S in the auxiliary variable feature E-Features ls , refers to the area of ​​the observed fermentation window; the strain area S in the auxiliary variable feature E-Features sp , refers to the total area of ​​the bacterial cluster observed in the observed fermentation window; the bacterial species spectral index I in the auxiliary variable feature E-Features sp , refers to the spectral index of the visible-near infrared I region to the near infrared II region of the bacterial mass in the observed fermentation window; the fermentation liquid spectral index I in the auxiliary variable feature E-Features ls , refers to the spectral index outside the visible-near infrared I region and near infrared II region of the fermentation liquid within the observed fermentation window.

6. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The terminal embedded intelligent devices in the S6 include: edge computing devices with independent graphics cards, built-in wireless network cards, SD card slots for data expansion, and earthquake-resistant, lightning-proof and anti-static designs for industrial-grade scenarios; the model deployment process includes: adopting unstructured pruning strategies to lightweight improve the machine vision model.

7. The soft measurement method of liquid bacterial biomass based on machine vision according to claim 1, characterized in that: The interactive software for executing the tasks of the cascaded machine vision soft-sensing model in S7 is a human-computer interface designed using pyQT6 to achieve the functions of interactive execution of machine vision tasks and cross-scenario transfer learning; The functions of the interactive software for machine vision task execution include: database expansion, model mobilization, model execution, model convergence training, model storage, model inference, and biomass parameter measurement effect display on the human-computer interaction interface; The cross-scene transfer learning of the cascaded machine vision soft measurement model MV refers to the generalization migration of the model through real-time interactive establishment of data sets, interactive model loading, and convergence training in cross-scene situations; the cross-scene application refers to the inference of target biomass parameters under different bacterial species and different field environments. If the dominant variables of the inferred biomass parameters do not meet the accuracy requirements, the data set is further expanded, and the model loading and convergence training are repeated until the accuracy requirements are met; if the inferred biomass parameters have met the accuracy requirements, it means that the model meets the online real-time non-invasive monitoring of biomass parameters in the current scene.

Images