Cell culture monitoring method based on fusion of near-infrared imaging and Raman spectroscopy

Abstract

The present invention discloses a cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy. Near-infrared image data and Raman spectroscopy data from the cell culture process are collected to predict cell culture monitoring indicators. The near-infrared image data and Raman spectroscopy data are normalized and block-encoded. A multi-head attention mechanism and a multi-head cross-attention mechanism are used to obtain corresponding near-infrared image initial features, Raman spectroscopy initial features, and initial mixed features. A multi-level progressive feature fusion is performed on the near-infrared image initial features, Raman spectroscopy initial features, and initial mixed features in three branches. After iterating the three branches according to the number of levels, the Raman spectroscopy fusion features finally output by the third branch are used to output the cell culture monitoring indicator prediction results. The present invention fully utilizes the high molecular weight specificity of Raman spectroscopy and the macroscopic spatial distribution advantages of near-infrared imaging to dynamically predict metabolite indicators during the cell culture process.

Description

Technical Field

[0001] The present invention relates to a cell culture monitoring method based on the fusion of near-infrared images and Raman spectra, belonging to the application of multimodal data fusion in cell culture monitoring. Background Art

[0002] Cell culture technology is a key enabling technology in modern biopharmaceuticals, tissue engineering, and biomedical research. Process stability and precise control are crucial for cell proliferation, differentiation, and the synthesis of target products. In practical production processes, such as vaccine preparation and antibody drug development, dynamic monitoring of key components during cell culture, such as nutrients (e.g., glucose and glutamine) and metabolites (e.g., lactate and ammonia), is crucial for process optimization and quality control.

[0003] Traditional cell culture process monitoring usually relies on offline sampling and chemical analysis methods, which are not only cumbersome and inefficient, but also the delayed feedback of offline detection after sampling will affect the real-time performance of dynamic cell culture control. Among the current online image monitoring technologies for cell culture process monitoring tasks, Raman spectroscopy is mostly used. With its high specificity for molecular vibrations of cells, Raman spectroscopy can accurately identify the structure and concentration changes of intracellular proteins, nucleic acids and metabolites. It is particularly suitable for high-sensitivity identification of specific chemical components. However, it lacks sensitivity and penetration for polar groups such as water, sugars and lipids produced by cell culture metabolism. In addition, the sampling points of Raman spectroscopy are discretely distributed, which cannot meet the characteristics of large-area component distribution detection in the cell culture process, making it difficult to capture the spatial macroscopic distribution of components in cell biological culture fluids.

[0004] For example, Chinese patent application number CN202210435377.4 discloses a real-time monitoring system and method for suspension cell culture. The system is used to monitor biochemical indicators and / or cellular changes in a suspension cell culture system within a bioreactor. The system comprises a data acquisition device and a data processing module. The data acquisition device includes a probe-type Raman spectrometer and / or an image acquisition device. The data processing module is configured to analyze Raman spectra and / or images acquired by the data acquisition device. The Raman spectral data analysis involves calculating quantitative biochemical indicator results from the real-time Raman spectral data using quantitative calibration models for the corresponding biochemical indicators. The image data analysis involves analyzing the real-time images using multiscale wavelet analysis to determine cell density and diameter distribution. This solution relies on Raman spectroscopy and partial least squares (PLS) for quantitative regression, making it difficult to address nonlinear variations caused by high concentrations, multiple components, and matrix interference. In complex cell culture systems, subtle changes in some modal data can be overlooked when culture conditions (such as pH, temperature) or feeding strategies change.

[0005] In addition, a single sensing technology often finds it difficult to balance the diversity of component types and detection sensitivity, and is unable to meet the comprehensive needs of high throughput, high dynamic range, and non-destructive real-time detection. Summary of the Invention

[0006] The technical problem solved by the present invention is: to address the problem that existing single sensing technology is difficult to take into account the diversity of cell culture metabolite component types and detection sensitivity, and to provide a cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy.

[0007] The present invention is implemented by the following technical solutions:

[0008] The present invention first discloses a cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy. The method collects near-infrared image data and Raman spectroscopy data during the cell culture process to predict the monitoring indicators of the cell culture. The prediction process includes the following steps:

[0009] S1. performing normalization processing and block coding on the near-infrared image data and the Raman spectrum data, and splicing explicit position coding on the block coding;

[0010] S2. The encoded near-infrared image data and Raman spectrum data are respectively subjected to a multi-head self-attention mechanism to obtain the corresponding near-infrared image initial features and Raman spectrum initial features, and then the near-infrared image initial features and Raman spectrum initial features are fused through a multi-head cross-attention mechanism to obtain the initial mixed features;

[0011] S3, divides the initial features of near-infrared image, initial features of Raman spectrum and initial mixed features into three branches to perform multi-level progressive feature fusion, among which,

[0012] The first branch and the third branch respectively extract the near-infrared image hierarchical features of the initial features of the near-infrared image and the Raman spectrum hierarchical features of the initial features of the Raman spectrum layer by layer through the multi-head self-attention mechanism.

[0013] The second branch passes the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted from the same layer together with the initial mixed features through the multi-modal spectral mixed attention model to obtain the mixed hierarchical features.

[0014] The third branch combines the mixed layer features and Raman spectrum layer features of the same layer through a multi-mode spectrum interaction fusion model to obtain Raman spectrum fusion features;

[0015] S4. After iterating the three branches of step S3 according to the number of levels, the monitoring index prediction result of cell culture is output through the Raman spectrum fusion feature finally output by the third branch.

[0016] In the cell culture monitoring method based on the fusion of near-infrared image and Raman spectrum of the present invention, further, in step S1, Z-score normalization is used to normalize the near-infrared image data and Raman spectrum data in the spectral dimension.

[0017] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy of the present invention, further, for near-infrared image data and Raman spectroscopy data with different data dimensions, the near-infrared image data encodes a three-dimensional image block at a time, and the Raman spectroscopy data encodes a segment of adjacent spectral curve values ​​at a time.

[0018] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy of the present invention, further, the explicit position encoding adopts a two-dimensional coordinate form to respectively encode the row and column positions of each image block in the near-infrared image data, and the row and column positions of the image blocks corresponding to the sampling points of the Raman spectral data on the near-infrared image. In the multimodal spectral mixed attention model in step S3 and the training process of the multimodal spectral mixed attention model, step S4 outputs the final near-infrared image data position encoding regression loss, Raman spectral data position encoding regression loss and prediction result regression loss of the first branch and the third branch, and the sum loss of the model is obtained by weighted summation.

[0019] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra of the present invention, further, the multi-modal spectral mixed attention model adopts an improved multi-head attention mechanism, and performs multi-head self-attention mechanism to calculate the initial mixed features with the near-infrared image layered features and the Raman spectrum layered features respectively to obtain near-infrared mixed features and Raman mixed features; the near-infrared mixed features, Raman mixed features and mixed features are input into the multi-head attention layer together, and the mixed layered features are output.

[0020] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra of the present invention, further, in the multimodal spectral interactive fusion module, the Raman spectrum layered features and the mixed layered features are subjected to dot product attention calculation to obtain a cross-correlation attention weight matrix, and the position encoding calculation corresponding to the Raman spectrum layered features is based on the Gaussian position attention matrix of the near-infrared image space. The cross-correlation attention weight matrix and the Gaussian position attention matrix are element-wise multiplied with the input mixed layered features, and a residual connection is performed with the Raman spectrum layer to output the Raman spectrum fusion features.

[0021] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy of the present invention, further, the Raman spectral hierarchical features and the mixed hierarchical features are both processed by the linear layer and then subjected to dot product attention calculation, and the cross-correlation attention weight matrix is ​​obtained by Softmax.

[0022] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra of the present invention, further, in the iteration of step S4, the near-infrared image hierarchical features extracted in the previous layer are used as the input of the multi-head self-attention mechanism of the next layer of the first branch, the mixed hierarchical features obtained in the previous layer and the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted in the next layer are used as the input of the multi-mode spectral mixed attention model of the next layer of the second branch, and the Raman spectrum fusion features obtained in the previous layer are used as the input of the multi-head self-attention mechanism of the next layer of the third branch.

[0023] The present invention also discloses a cell culture monitoring system based on the fusion of near-infrared images and Raman spectra, comprising an acquisition module, a data processing module and a data fusion prediction module.

[0024] The acquisition module collects near-infrared image data and Raman spectrum data during the cell culture process.

[0025] The data processing module normalizes and block encodes the near-infrared image data and Raman spectrum data, splices explicit position coding on the block coding, and obtains the corresponding near-infrared image initial features and Raman spectrum initial features from the encoded near-infrared image data and Raman spectrum data respectively through a multi-head self-attention mechanism, and fuses the near-infrared image initial features and the Raman spectrum initial features through a multi-head cross-attention mechanism to obtain initial mixed features.

[0026] The data fusion prediction module adopts a three-branch multi-level structure to perform multi-level progressive feature fusion on the initial features of the near-infrared image, the initial features of the Raman spectrum and the initial mixed features. Among them, the first branch and the third branch take the initial features of the near-infrared image and the initial features of the Raman spectrum as input respectively. Multi-head self-attention layers are set layer by layer on the first branch and the third branch to extract the near-infrared image hierarchical features and the Raman spectrum hierarchical features layer by layer. The second branch takes the initial mixed features and the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted at the same layer as input. The second branch connects the multi-mode spectral mixed attention model layer by layer. The multi-mode spectral mixed attention model outputs the mixed hierarchical features layer by layer. The mixed hierarchical features obtained in the first layer and the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted in the next layer are used as the input of the multi-modal spectral mixed attention model of the next layer of the second branch. The third branch is also connected to the multi-modal spectral interaction fusion model after each multi-head self-attention layer, with the mixed hierarchical features and Raman spectrum hierarchical features of the same layer as input. The multi-modal spectral interaction fusion model outputs Raman spectrum fusion features layer by layer, and the Raman spectrum fusion features obtained in the previous layer are used as the input of the multi-head self-attention mechanism of the next layer of the third branch. The multi-modal spectral interaction fusion model of the last layer of the third branch is connected to the output layer, and finally outputs the monitoring indicators of cell culture predicted by the Raman spectrum fusion features through the third branch.

[0027] The present invention also discloses a computer terminal, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it can be used to execute the above-mentioned cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy of the present invention.

[0028] The proposed cell culture monitoring method, based on the fusion of near-infrared images and Raman spectroscopy, differs from conventional cell culture process monitoring methods that use a single Raman spectrum. By fusing the local information of a single-point Raman spectrum with the global information of a near-infrared image, this method predicts essential monitoring indicators during the cell culture process (such as cell viability, lactate concentration, and ammonium nitrogen concentration). To address the dimensionality and modality mismatch between near-infrared image data and Raman spectroscopy data, the present invention designs a three-branch multi-level progressive fusion framework, combining a multimodal spectral hybrid attention model and a multimodal spectral interactive fusion model to fuse near-infrared image and Raman spectroscopy data.

[0029] The present invention adopts the above technical solution to achieve the following beneficial effects:

[0030] (1) The present invention utilizes the sensitivity and strong penetration ability of near-infrared imaging to polar groups such as water, sugars and lipids, as well as the ability to detect large-area component distribution, to simultaneously collect near-infrared image data and Raman spectral data during the cell culture process, and combines the multi-level progressive fusion framework with a three-branch structure to fuse the near-infrared image data and Raman spectral data, providing sufficient buffer for the fusion of multimodal features, gradually narrowing the differences in distribution and expression between different modalities, and alleviating the problems of fusion failure and information bias caused by strong modal heterogeneity in traditional direct fusion methods, thereby improving the robustness and expressiveness of the fusion features of near-infrared data and Raman spectral data, obtaining clearer predictive biological indicators in the cell culture process, and being able to quickly capture the spatial distribution of components in the culture medium on a macro scale, which is suitable for obtaining changes in the overall properties of the sample.

[0031] (2) This invention introduces a multimodal spectral hybrid attention model. The hybrid attention mechanism can dynamically adjust the modal weights, increase the attention to one modality when the signal of another modality is not obvious, and highlight key features that are more sensitive to abnormal metabolic changes (such as sudden lactate accumulation or glutamine depletion). In addition, the semantic dependency relationship between near-infrared image data and Raman spectral data is modeled globally, and the attention weights of different modal information are dynamically adjusted, thereby effectively extracting key features that are more meaningful for discriminating target metabolic components in cell culture processes.

[0032] (3) The design of the present invention also adopts a multi-mode spectral interactive fusion model to enhance the complementarity between different information sources of near-infrared image data and Raman spectral data, and fully utilize the differences and complementary characteristics of the two modalities in the spectral response mechanism. Near-infrared image data mainly reflects the broadband absorption characteristics based on intermolecular vibrations, and is good at capturing the overall change trend of macromolecular components such as water, protein, and lipids, and can characterize the dynamic changes of cell density, biomass growth, and nutrients; while Raman spectral data provides high-resolution structural information based on intramolecular vibration modes, and is particularly sensitive to chemical bond types, functional group configurations, and molecular skeleton characteristics. It is more suitable for revealing the structural changes and concentration fluctuations of key metabolites such as glucose, glutamine, lactic acid, and ammonia. This complementarity between modalities allows for a simple and efficient feature fusion of near-infrared image features and Raman spectral features, allowing the fusion model to synergistically perceive changes in culture status from both the macroscopic cell behavior and microscopic metabolic levels, thereby achieving more comprehensive and accurate process monitoring of the cell culture process.

[0033] (4) In the process of processing near-infrared image data and Raman spectrum data, the present invention introduces an explicit two-dimensional position coding mechanism, and combines the position coding loss to perform spatial constraints during the model training process, so that the model can accurately perceive the spatial correspondence between the Raman spectrum sampling points in the near-infrared image. In the process of model training, the explicit position coding loss of the near-infrared image feature branch and the Raman spectrum feature branch are used to enhance the model's understanding of the implicit geometric constraints between spatial structure and modality, construct a more physically reasonable cross-modal fusion mechanism, and improve the accuracy of dynamic monitoring and prediction of the cell culture process.

[0034] In summary, the cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy provided by the present invention collects Raman spectral data and near-infrared image data during the cell culture process, and uses a deep learning method to realize the multimodal spectral feature fusion of the two data. It fully utilizes the high molecular specificity of Raman spectroscopy and the macroscopic spatial distribution advantages of near-infrared imaging to dynamically predict the metabolite indicators in the cell culture process, meeting the higher requirements of modern biomanufacturing for the real-time, stability and generalization capabilities of process analysis technology (PAT).

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the process of fusing near-infrared image data and Raman spectroscopy data in the cell culture monitoring method of the embodiment.

[0037] Figure 2 Schematic diagram of the multi-modal spectral mixed attention model at the first level of the second branch in an embodiment.

[0038] Figure 3 Schematic diagram of the multi-mode spectrum interaction fusion model at the first level of the third branch in the embodiment.

[0039] Figure 4 4 is a structural block diagram of a cell culture monitoring system based on the fusion of near-infrared images and Raman spectroscopy according to an embodiment. DETAILED DESCRIPTION

[0040] See also Figure 1 This embodiment provides a cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy. During the cell culture process, near-infrared image data and Raman spectroscopy data of the cell culture environment are collected to predict cell culture monitoring indicators, including but not limited to cell activity, lactate concentration, ammonium nitrogen concentration, and other monitoring indicators. The specific prediction process is as follows:

[0041] S1. Perform normalization processing and block coding on the near-infrared image data and Raman spectrum data, and splice explicit position coding on the block coding.

[0042] Z-score normalization is used to normalize the near-infrared image data and Raman spectrum data in the spectral dimension. Specifically, the formulas for Z-score normalization of the near-infrared image data NI and the Raman spectrum data RM are as follows: ;in is the near-infrared image data NI or Raman spectrum data RM to be normalized, is the mean of the corresponding data in the corresponding spectral channel dimension, is the variance of the corresponding data in the corresponding spectral channel dimension.

[0043] The normalized near-infrared image data and Raman spectrum data are encoded in patches, resulting in near-infrared image and Raman spectrum encodings, with explicit position encoding added. For near-infrared image data and Raman spectrum data with different data dimensions, near-infrared image encoding uses a three-dimensional image patch as a patch, while Raman spectrum encoding uses a small segment of adjacent spectral curve values ​​as a patch. Explicit position encoding uses two-dimensional coordinates to encode the row and column positions of each image patch in the near-infrared image data, as well as the row and column positions of the image patches corresponding to the sampling points of the Raman spectrum data on the near-infrared image.

[0044] Specifically, for the normalized near-infrared image data With Raman spectroscopy data Use two-dimensional convolution and one-dimensional convolution to encode the patches and obtain the near-infrared image encoding. and Raman spectral encoding ,in and Indicates the number of patches. Represents the dimension of the feature. A single Patch of near-infrared image encoding corresponds to image blocks, is the set block length, is the number of bands of the near-infrared image; a single patch of Raman spectral data corresponds to the adjacent spectral curve values, the size is , is the number of samples of Raman spectrum data during sampling, is the number of adjacent spectral curve values ​​​​encoded, is the number of bands in the Raman spectrum data.

[0045] Encoding near-infrared images Encoding with Raman spectroscopy Introduce explicit position coding to express the absolute position of each patch on the original image. For example, the position coding of the first row and first column of the patch is (1,1), and it is spliced ​​to the beginning of the corresponding feature to obtain the near-infrared image feature with position coding. The position encoding of the Raman spectrum is determined by the position of the pixel corresponding to the near-infrared image. The position of the patch to which the pixel belongs is used as the position encoding to obtain the Raman spectrum feature with position information. If there are multiple Raman spectrum sampling points, the coordinate value is determined by the center position. The center position calculation formula is: , is the center position of n Raman spectrum sampling points, is the position of the i-th Raman spectrum sampling point, that is, the horizontal and vertical coordinate values ​​of multiple sampling points are averaged to obtain the explicit position code (m,n) of the corresponding Raman spectrum feature.

[0046] S2. The encoded near-infrared image data and Raman spectrum data are respectively subjected to a multi-head self-attention mechanism to obtain the corresponding near-infrared image initial features and Raman spectrum initial features, and the near-infrared image initial features and the Raman spectrum initial features are fused through a multi-head cross-attention mechanism to obtain the initial mixed features.

[0047] Specifically, the near-infrared image features obtained by encoding in step 1 Raman spectral characteristics The initial features of the near-infrared image are obtained by using the multi-head self-attention layer in the original Transformer. and initial Raman spectral features Then, the initial features of the near-infrared image are used as the key vector Key and the value vector Value, and the initial features of the Raman spectrum are used as the query vector Quary, which are input into the multi-head cross attention layer of the original Transformer to output the initial mixed features. The multi-head self-attention layer and the multi-head cross-attention layer not only include the calculation of the attention mechanism, but also include feedforward networks and normalization, etc. The calculation of the attention mechanism belongs to the well-known neural network calculation method. This embodiment does not elaborate on the specific attention mechanism calculation process.

[0048] S3, divides the initial features of near-infrared images, initial features of Raman spectra and initial mixed features into three branches for multi-level progressive feature fusion.

[0049] The first branch and the third branch respectively extract the near-infrared image hierarchical features of the initial features of the near-infrared image and the Raman spectrum hierarchical features of the initial features of the Raman spectrum layer by layer through a multi-layer multi-head self-attention mechanism. The near-infrared image hierarchical features extracted from the previous layer serve as the input of the multi-head self-attention mechanism in the next layer of the first branch.

[0050] The second branch passes the near-infrared image layered features and Raman spectrum layered features extracted from the same layer together with the initial mixed features through a multi-modal spectral mixed attention model. The multi-modal spectral mixed attention model uses an improved multi-head attention mechanism. The initial mixed features are calculated by performing dot product attention matrices with the near-infrared image layered features and Raman spectrum layered features, respectively. The attention matrices are then element-wise multiplied with the corresponding near-infrared image layered features or Raman spectrum layered features to obtain near-infrared mixed features and Raman mixed features, respectively. The near-infrared mixed features and Raman mixed features are used together with the mixed features as the input of the multi-head attention layer, where the mixed features serve as the query vector Quary, the near-infrared mixed features serve as the key vector Key, and the Raman mixed features serve as the value vector Value. The mixed hierarchical features are output from the multi-head attention layer. The mixed hierarchical features obtained from the previous layer and the near-infrared image layered features and Raman spectrum layered features extracted from the next layer serve as the input of the multi-modal spectral mixed attention model in the next layer of the second branch.

[0051] The specific calculation process is as follows Figure 2 As shown, in the multi-modal spectral mixed attention model at the first level of the second branch, the initial mixed feature And the initial features of the near-infrared image calculated by the second multi-head self-attention mechanism And the initial characteristics of Raman spectrum Input into the multimodal spectral mixed attention model. The multimodal spectral mixed attention model first converts the initial mixed features The initial features of the near-infrared image calculated by the second multi-head self-attention mechanism And the initial characteristics of Raman spectrum Calculate the attention separately to obtain the near-infrared mixed features and Raman mixing characteristics , and the calculation formulas are as follows:

[0052] 、

[0053] .

[0054] in is the scaling factor, and T represents the matrix transpose.

[0055] Then the near infrared mixed feature Mixed with Raman characteristics Together as the input of the multi-head attention layer, where the initial mixed features As the query vector Quary, near-infrared mixed features As the key vector Key, Raman mixing feature As the value vector Value, its output is the mixed hierarchical feature As the input of the next layer of the second branch. The calculation formula of this process is as follows:

[0056] 、

[0057] .

[0058] in is the scaling factor, is the output of attention calculation, is the layer normalization operation, It is the feedforward network layer in the original Transformer, and T represents matrix transpose.

[0059] The third branch integrates the mixed layer features and Raman spectrum layer features of the same layer through a multimodal spectral interaction fusion model. This model performs dot product attention on the mixed layer features processed by the linear layer and the Raman spectrum layer features, and then performs a softmax operation to obtain a cross-correlation attention weight matrix. A Gaussian position attention matrix based on the position encoding corresponding to the Raman spectrum layer features is calculated in the near-infrared image space. After the Gaussian position attention matrix is ​​processed by the linear layer, it is element-wise multiplied with the cross-correlation attention weight matrix and the input mixed layer features to obtain the Raman spectrum fusion feature. The Raman spectrum fusion feature obtained in the previous layer serves as the input to the multi-head self-attention mechanism in the next layer of the third branch.

[0060] like Figure 3As shown, in the multimodal spectral interaction fusion model of the first level of the third branch, the mixed hierarchical features output by the multimodal spectral mixed attention model of the same layer as the second branch are , and the initial Raman spectrum features calculated by the third branch through the multi-head self-attention mechanism The multi-mode spectral interaction fusion model firstly combines the mixed hierarchical features The initial Raman spectrum features calculated by the third branch through the multi-head self-attention mechanism After being processed by the linear layer, the attention weight matrix is ​​obtained by the attention calculation. , the process is as follows:

[0061] .

[0062] in represents a linear layer, is the scaling factor.

[0063] Then, according to the explicit position encoding (m, n) of the Raman spectral feature, combined with the near-infrared image coordinates, the Gaussian position attention matrix based on the near-infrared image space is calculated. , the calculation formula of Gaussian position attention is as follows:

[0064] .

[0065] in, Refers to the Gaussian position attention matrix One of the elements, and They correspond to the horizontal and vertical coordinate positions of a certain patch of near-infrared image features, represents the natural exponential operation, Indicates the degree of diffusion of the Gaussian kernel, which is an adjustable hyperparameter. The position coordinates of all patches in the near-infrared feature are calculated to obtain the corresponding Gaussian position attention, forming a Gaussian position attention matrix. .

[0066] Then, the obtained Gaussian position attention matrix After a learnable linear transformation matrix Processing, and then with the cross-correlation attention weight matrix Multiply element by element to get the fusion matrix ; Fusion matrix With mixed hierarchical features Multiply them together to get the fusion feature, and finally add it to the initial Raman spectrum feature calculated by the multi-head self-attention mechanism of the third branch Perform residual connection and output Raman spectrum fusion features , the process is as follows:

[0067] .

[0068] in Represents element-wise multiplication.

[0069] The Raman spectrum fusion features output by the multi-mode spectrum interaction fusion model of this layer , as the input of the multi-head self-attention layer of the next layer of the third branch, and output new Raman spectrum fusion features layer by layer.

[0070] S4: After iterating the three branches of step S3 for N layers, the Raman spectral fusion features finally output by the third branch are used to output the predicted results of the cell culture monitoring indicators. N is the number of layers in the three branches. The range of N can be based on actual engineering experience. The initial number of layers N (for example, N = 6) is selected. During training, the model accuracy and hardware resource usage are then monitored. If hardware resources are insufficient or the accuracy is high, 1-3 layers can be appropriately reduced. If the training accuracy is lower than the required accuracy and the hardware resources are sufficient, 1-3 layers can be appropriately added. Repeat the adjustment until the model meets the actual accuracy requirements. More layers increase the hardware (graphics card) requirements and the training and inference time costs. Therefore, the number of layers should be as small as possible while meeting the accuracy requirements.

[0071] The above-mentioned multimodal spectral mixed attention model and multimodal spectral mixed attention model are trained according to the above steps by collecting multiple sets of near-infrared image data and multi-sampling point Raman spectral data during the cell culture process, and the cell culture monitoring index results analyzed offline during the near-infrared image data and Raman spectral data collection period are used as labels for the corresponding training data. During the training process of the multimodal spectral mixed attention model and the multimodal spectral mixed attention model, step S4 outputs the final near-infrared image data position encoding regression loss, Raman spectral data position encoding regression loss and prediction result regression loss of the first branch and the third branch, and the sum loss of the model is obtained by weighted summation. In the actual cell culture monitoring process, the trained multimodal spectral mixed attention model and the multimodal spectral mixed attention model directly predict the monitoring indicators in the cell culture process.

[0072] That is to say, during the model training process, the near-infrared hierarchical features of the first and third branches are finally obtained. and Raman spectroscopy fusion features , by the final near-infrared layered features Fusion features with the final Raman spectrum Calculate the position encoding regression loss of the first two features to obtain the position encoding regression loss of the near-infrared image data and Raman spectral data position encoding regression loss The position encoding regression loss function uses MSE loss, and the loss calculation formula is as follows:

[0073] .

[0074] in and Indicates the horizontal and vertical coordinate values ​​of the initial position coding of the near-infrared image feature and the initial position coding of the Raman spectrum feature, and To output the horizontal and vertical coordinate values ​​of the position encoding in the final near-infrared layering and Raman spectral characteristics, Indicates the number of features.

[0075] According to the Raman spectrum fusion characteristics of the final output of the third branch , after processing by the output layer, the prediction results of cell culture monitoring indicators are obtained , K is the number of monitoring indicators; the output layer consists of full connection and activation function, and its process formula is as follows:

[0076] .

[0077] in is a learnable linear transformation matrix, , is the bias term, is the activation function.

[0078] Combined with the actual monitoring index results of cell culture analyzed offline during the near-infrared image data and Raman spectroscopy data collection period, the prediction results are calculated The prediction regression loss , the loss function also uses MSE loss, and the prediction result loss calculation method is position coding regression loss calculation. Then the near infrared image data position coding regression loss , Raman spectral data position encoding regression loss And the prediction result regression loss Weighted combination gain and loss , used for model training, and loss The calculation is as follows:

[0079] .

[0080] in is a configurable hyperparameter, .

[0081] See also Figure 4This embodiment also discloses a cell culture monitoring system based on the fusion of near-infrared images and Raman spectroscopy, including an acquisition module 100, a data processing module 200 and a data fusion prediction module 300.

[0082] The acquisition module 100 collects near-infrared image data and Raman spectrum data during the cell culture process, including an image acquisition device for collecting near-infrared images and a spectral instrument for sampling Raman spectra. These are conventional equipment for cell culture detection. This embodiment does not elaborate on the specific methods for collecting near-infrared image data and Raman spectrum data.

[0083] The data processing module 200 normalizes and block encodes the near-infrared image data and Raman spectrum data, splices explicit position coding on the block coding, and obtains the corresponding near-infrared image initial features and Raman spectrum initial features from the encoded near-infrared image data and Raman spectrum data through a multi-head attention mechanism, and fuses the near-infrared image initial features and the Raman spectrum initial features through a multi-head cross-attention mechanism to obtain initial mixed features.

[0084] The data fusion prediction module 300 adopts a three-branch, multi-level structure to perform multi-level progressive feature fusion on the initial features of the near-infrared image, the initial features of the Raman spectrum, and the initial mixed features. The first branch 310 and the third branch 330 take the initial features of the near-infrared image and the initial features of the Raman spectrum as input, respectively. Multi-head self-attention layers are set layer by layer on the first branch 310 and the third branch 330 to extract the near-infrared image layer features and the Raman spectrum layer features layer by layer. The second branch 320 takes the initial mixed features and the near-infrared image layer features and Raman spectrum layer features extracted at the same layer as input. The second branch 320 connects the trained multimodal spectral mixed attention model layer by layer, and outputs the mixed layer features layer by layer through the multimodal spectral mixed attention model. The mixed layer features obtained in the previous layer and the near-infrared image layer features and Raman spectrum layer features extracted in the next layer serve as the input to the multimodal spectral mixed attention model in the next layer of the second branch 320. The third branch 330 connects to a trained multimodal spectral interaction fusion model after each multi-head self-attention layer. Taking the mixed layer features and Raman spectral layer features of the same layer as input, the multimodal spectral interaction fusion model outputs Raman spectral fusion features layer by layer. The Raman spectral fusion features obtained in the previous layer serve as the input to the multi-head self-attention mechanism in the next layer of the third branch. The multimodal spectral interaction fusion model in the last layer of the third branch 330 is connected to the output layer, and the third branch 330 ultimately outputs the cell culture monitoring indicators predicted by the Raman spectral fusion features.

[0085] This embodiment also discloses a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the terminal can be used to execute the cell culture monitoring method based on the fusion of near-infrared images and Raman spectroscopy described above in this embodiment.

[0086] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0087] In this document, the directions or positional relationships indicated by terms such as "up", "down", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "vertical", and "horizontal" are based on the directions or positional relationships shown in the accompanying drawings and are only for the clarity of the technical solution and the convenience of description, and therefore should not be understood as limiting the present invention.

[0088] As used herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion of elements other than the listed elements and may also include additional elements not specifically listed.

[0089] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy, characterized by: Near-infrared image data and Raman spectroscopy data collected during cell culture are used to predict monitoring indicators of cell culture. The prediction process includes the following steps: S1. performing normalization processing and block coding on the near-infrared image data and the Raman spectrum data, and splicing explicit position coding on the block coding; S2. The encoded near-infrared image data and Raman spectrum data are respectively subjected to a multi-head self-attention mechanism to obtain the corresponding near-infrared image initial features and Raman spectrum initial features, and then the near-infrared image initial features and Raman spectrum initial features are fused through a multi-head cross-attention mechanism to obtain the initial mixed features; S3, divides the initial features of near-infrared image, initial features of Raman spectrum and initial mixed features into three branches to perform multi-level progressive feature fusion, among which, The first branch and the third branch respectively extract the near-infrared image hierarchical features of the initial features of the near-infrared image and the Raman spectrum hierarchical features of the initial features of the Raman spectrum layer by layer through the multi-head self-attention mechanism. The second branch passes the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted from the same layer together with the initial mixed features through the multi-modal spectral mixed attention model to obtain the mixed hierarchical features. The third branch combines the mixed layer features and Raman spectrum layer features of the same layer through a multi-mode spectrum interaction fusion model to obtain Raman spectrum fusion features; S4. After iterating the three branches of step S3 according to the number of levels, the monitoring index prediction result of cell culture is output through the Raman spectrum fusion feature finally output by the third branch.

2. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 1, characterized in that: In step S1, Z-score normalization is used to perform normalization processing on the near-infrared image data and the Raman spectrum data in the spectral dimension.

3. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 2, characterized in that: For near-infrared image data and Raman spectrum data with different data dimensions, the near-infrared image data encodes a three-dimensional image block at a time, and the Raman spectrum data encodes a segment of adjacent spectral curve values ​​at a time.

4. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 1, characterized in that: The explicit position encoding adopts a two-dimensional coordinate form to respectively encode the row and column positions of each image block in the near-infrared image data, and the row and column positions of the image blocks corresponding to the sampling points of the Raman spectrum data on the near-infrared image. During the multi-modal spectral mixed attention model in step S3 and the training process of the multi-modal spectral mixed attention model, step S4 outputs the final near-infrared image data position encoding regression loss, Raman spectrum data position encoding regression loss and prediction result regression loss of the first branch and the third branch, and the sum loss of the model is obtained by weighted summation.

5. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 1, characterized in that: The multi-modal spectral mixed attention model adopts an improved multi-head attention mechanism, and performs multi-head self-attention mechanism on the initial mixed features, the near-infrared image layered features and the Raman spectrum layered features to obtain near-infrared mixed features and Raman mixed features; the near-infrared mixed features, Raman mixed features and mixed features are input into the multi-head attention layer together, and the mixed layered features are output.

6. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 1, characterized in that: In the multimodal spectral interactive fusion module, the Raman spectrum layered features and the mixed layered features are subjected to dot product attention calculation to obtain a cross-correlation attention weight matrix. The position encoding calculation corresponding to the Raman spectrum layered features is based on the Gaussian position attention matrix in the near-infrared image space. The cross-correlation attention weight matrix and the Gaussian position attention matrix are element-wise multiplied with the input mixed layered features, and a residual connection is performed with the Raman spectrum layer to output the Raman spectrum fusion features.

7. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 6, characterized in that: The Raman spectrum hierarchical features and the mixed hierarchical features are both processed by the linear layer and then the dot product attention calculation is performed, and the cross-correlation attention weight matrix is ​​obtained by Softmax.

8. The cell culture monitoring method based on the fusion of near-infrared imaging and Raman spectroscopy according to claim 1, characterized in that: In the iteration of step S4, the near-infrared image hierarchical features extracted in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the first branch, the mixed hierarchical features obtained in the previous layer and the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted in the next layer are used as the input of the multi-mode spectrum mixed attention model in the next layer of the second branch, and the Raman spectrum fusion features obtained in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the third branch.

9. A cell culture monitoring system based on the fusion of near-infrared imaging and Raman spectroscopy is characterized by include: An acquisition module collects near-infrared image data and Raman spectral data during the cell culture process; a data processing module that normalizes and block-encodes the near-infrared image data and Raman spectrum data, splices explicit position codes on the block codes, and uses a multi-head self-attention mechanism to obtain corresponding near-infrared image initial features and Raman spectrum initial features from the encoded near-infrared image data and Raman spectrum data, and fuses the near-infrared image initial features and Raman spectrum initial features with the multi-head cross-attention mechanism to obtain initial mixed features; The data fusion prediction module adopts a three-branch multi-level structure to perform multi-level progressive feature fusion on the initial features of the near-infrared image, the initial features of the Raman spectrum, and the initial mixed features. The first branch and the third branch take the initial features of the near-infrared image and the initial features of the Raman spectrum as input respectively. Multi-head self-attention layers are set layer by layer on the first branch and the third branch to extract the hierarchical features of the near-infrared image and the hierarchical features of the Raman spectrum layer by layer. The second branch takes the initial mixed features and the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted from the same layer as input. The second branch connects the multimodal spectral mixed attention model layer by layer. The multimodal spectral mixed attention model outputs the mixed hierarchical features layer by layer. The mixed hierarchical features obtained in the previous layer and the near-infrared image hierarchical features and Raman spectrum hierarchical features extracted in the next layer are used as the input of the multimodal spectral mixed attention model in the next layer of the second branch. The third branch is also connected to a multi-mode spectral interaction fusion model after each multi-head self-attention layer. It takes the mixed layer features and Raman spectrum layer features of the same layer as input. The multi-mode spectral interaction fusion model outputs Raman spectrum fusion features layer by layer. The Raman spectrum fusion features obtained in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the third branch. The multi-mode spectral interaction fusion model at the last layer of the third branch is connected to the output layer, and the monitoring indicators of cell culture predicted by the Raman spectral fusion features are finally output through the third branch.

10. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 8.

Images