Cell culture monitoring method based on fusion of near infrared image and Raman spectrum

Abstract

The invention discloses a cell culture monitoring method based on fusion of a near infrared image and a Raman spectrum, which comprises the following steps of: acquiring near infrared image data and Raman spectrum data in a cell culture process to predict monitoring indexes of cell culture, and performing normalization processing and block coding on the near infrared image data and the Raman spectrum data; corresponding near-infrared image initial features, Raman spectrum initial features and initial mixed features are obtained through a multi-head attention mechanism and a multi-head cross attention mechanism; and carrying out multi-level progressive feature fusion on the near-infrared image initial features, the Raman spectrum initial features and the initial mixed features in three branches, carrying out iteration on the three branches according to the level number, and outputting a monitoring index prediction result of cell culture through the Raman spectrum fusion features finally output by the third branch. According to the method, the metabolite indexes in the cell culture process are dynamically predicted by fully utilizing the macroscopic spatial distribution advantages of the polymer specificity of the Raman spectrum and the near-infrared imaging.

Description

Technical Field

[0001] The present invention relates to a method for monitoring cell culture based on the fusion of near-infrared images and Raman spectra, belonging to the application of multimodal data fusion in cell culture monitoring. Background Art

[0002] Cell culture technology is a key supporting technology in modern biopharmaceuticals, tissue engineering, and biomedical research. The stability and precise control of its process are of great significance for cell proliferation, differentiation, and the synthesis of target products. In the actual production process, such as vaccine preparation and antibody drug development, the dynamic monitoring of key components such as nutrients (such as glucose, glutamine) and metabolites (such as lactate, ammonia) in the cell culture process is crucial for process optimization and quality control.

[0003] Traditional cell culture process monitoring usually relies on off-line sampling and chemical analysis methods, which are not only cumbersome and inefficient in operation, but also the delayed feedback of off-line detection after sampling will affect the real-time performance of cell culture dynamic regulation. In the current online image monitoring technology for cell culture process monitoring tasks, Raman spectroscopy is mostly used. Raman spectroscopy, with its high specificity for the molecular vibrations of cells, can accurately identify the structural and concentration changes of proteins, nucleic acids, and metabolites in cells, and is particularly suitable for highly sensitive identification of specific chemical components. However, its sensitivity and penetration ability for polar groups such as water, sugars, and lipids produced by cell culture metabolism are insufficient, and the sampling points of Raman spectroscopy are discretely distributed, which cannot meet the characteristics of large-area component distribution detection in the cell culture process, and it is difficult to capture the spatial macroscopic distribution of components in cell biological culture media.

[0004] For example, Chinese Patent Application No. CN202210435377.4 discloses a real-time monitoring system and method for suspension cell culture, which is used to monitor the biochemical indexes and / or cell changes in the suspension cell culture system in a bioreactor in real time, including a data acquisition device and a data processing module; the data acquisition device includes a probe-type Raman spectrometer and / or an image acquisition device; the data processing module is used to perform data analysis on the Raman spectra and / or images collected by the data acquisition device; the data analysis of Raman spectra includes calculating the quantitative results of biochemical indexes in real time through a quantitative calibration model of corresponding biochemical indexes for the Raman spectrum data collected in real time; the image data analysis includes obtaining the density and diameter distribution results of cells through a multi-scale wavelet analysis method for the images collected in real time. This solution relies on Raman spectroscopy and partial least squares (PLS) for quantitative regression, and it is difficult to cope with the non-linear changes brought about by high concentrations, multi-components, and matrix interference. In a complex cell culture system, when culture conditions (such as pH, temperature) or feeding strategies change, the weak changes in some modal data may be ignored.

[0005] In addition, a single sensing technology often has difficulty in balancing the diversity of component types and detection sensitivity, and it is difficult to meet the comprehensive requirements of high-throughput, high dynamic range, and non-destructive real-time detection. Summary of the Invention

[0006] The technical problem solved by the present invention is: aiming at the problem that the existing single sensing technology is difficult to balance the diversity of component types of cell culture metabolites and detection sensitivity, a cell culture monitoring method based on the fusion of near-infrared images and Raman spectra is provided.

[0007] The present invention is implemented by adopting the following technical solutions:

[0008] The present invention first discloses a cell culture monitoring method based on the fusion of near-infrared images and Raman spectra, which collects near-infrared image data and Raman spectrum data during the cell culture process to predict the monitoring indicators of cell culture. The prediction process includes the following steps:

[0009] S1. Normalize and block-encode the near-infrared image data and Raman spectrum data, and splice explicit position encoding on the block encoding;

[0010] S2. Respectively pass the encoded near-infrared image data and Raman spectrum data through the multi-head self-attention mechanism to obtain the corresponding initial near-infrared image features and initial Raman spectrum features, and then fuse the initial near-infrared image features and initial Raman spectrum features through the multi-head cross-attention mechanism to obtain the initial mixed features;

[0011] S3. Perform multi-level progressive feature fusion on the initial near-infrared image features, initial Raman spectrum features, and initial mixed features in three branches. Among them,

[0012] The first branch and the third branch respectively extract the hierarchical near-infrared image features of the initial near-infrared image features and the hierarchical Raman spectrum features of the initial Raman spectrum features layer by layer through the multi-head self-attention mechanism.

[0013] The second branch passes the hierarchical near-infrared image features and hierarchical Raman spectrum features extracted in the same layer together with the initial mixed features through the multi-modal spectral mixture attention model to obtain the mixed hierarchical features.

[0014] The third branch passes the mixed hierarchical features and hierarchical Raman spectrum features in the same layer through the multi-modal spectral interaction fusion model to obtain the Raman spectrum fusion features;

[0015] S4. After iterating the three branches in step S3 according to the number of hierarchical levels, output the prediction result of the cell culture monitoring index through the Raman spectrum fusion features finally output by the third branch.

[0016] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, in step S1, Z-score normalization is used to perform normalization processing on the near-infrared image data and the Raman spectrum data in the spectral dimension.

[0017] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, for the near-infrared image data and the Raman spectrum data with different data dimensions, the near-infrared image data encodes a three-dimensional image block at a time, and the Raman spectrum data encodes a segment of adjacent spectral curve values at a time.

[0018] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, the explicit position encoding adopts a two-dimensional coordinate form to encode the row and column positions of each image block in the near-infrared image data, and the row and column positions of the sampling points of the Raman spectrum data corresponding to the image blocks on the near-infrared image. In the multi-modal spectral hybrid attention model and the training process of the multi-modal spectral hybrid attention model in step S3, the near-infrared image data position encoding regression loss, the Raman spectrum data position encoding regression loss, and the prediction result regression loss are output from step S4, and the sum loss of the model is obtained by weighted summation.

[0019] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, the multi-modal spectral hybrid attention model adopts an improved multi-head attention mechanism to perform multi-head self-attention mechanism calculations on the initial hybrid features with the near-infrared image hierarchical features and the Raman spectrum hierarchical features respectively to obtain near-infrared hybrid features and Raman hybrid features; the near-infrared hybrid features, the Raman hybrid features, and the hybrid features are input into the multi-head attention layer together to output hybrid hierarchical features.

[0020] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, in the multi-modal spectral interaction and fusion module, the Raman spectrum hierarchical features and the hybrid hierarchical features perform dot product attention calculation to obtain a cross-correlation attention weight matrix, and the position encoding corresponding to the Raman spectrum hierarchical features calculates a Gaussian position attention matrix based on the near-infrared image space. The cross-correlation attention weight matrix is multiplied element-wise with the Gaussian position attention matrix and the input hybrid hierarchical features, and a residual connection is made with the Raman spectrum hierarchical features to output Raman spectrum fusion features.

[0021] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, both the Raman spectrum hierarchical features and the hybrid hierarchical features are processed by a linear layer and then perform dot product attention calculation, and the cross-correlation attention weight matrix is obtained through Softmax.

[0022] In the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to the present invention, further, in the iteration of step S4, the hierarchical features of the near-infrared images extracted in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the first branch, and the mixed hierarchical features obtained in the previous layer, the hierarchical features of the near-infrared images and the hierarchical features of the Raman spectra extracted in the next layer are used as the input of the multi-modal spectral mixing attention model in the next layer of the second branch, and the Raman spectrum fusion features obtained in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the third branch.

[0023] The present invention also discloses a cell culture monitoring system based on the fusion of near-infrared images and Raman spectra, including an acquisition module, a data processing module, and a data fusion prediction module.

[0024] The acquisition module acquires near-infrared image data and Raman spectrum data during the cell culture process.

[0025] The data processing module performs normalization processing and block coding on the near-infrared image data and Raman spectrum data, splices explicit position coding on the block coding, and respectively passes the coded near-infrared image data and Raman spectrum data through a multi-head self-attention mechanism to obtain corresponding initial features of the near-infrared images and initial features of the Raman spectra, and fuses the initial features of the near-infrared images and the initial features of the Raman spectra through a multi-head cross-attention mechanism to obtain initial mixed features.

[0026] The data fusion prediction module adopts a three-branch multi-level structure to perform multi-level progressive feature fusion on the initial features of the near-infrared images, the initial features of the Raman spectra, and the initial mixed features. Among them, the first branch and the third branch respectively use the initial features of the near-infrared images and the initial features of the Raman spectra as inputs. The first branch and the third branch are sequentially provided with multi-head self-attention layers, and hierarchical features of the near-infrared images and hierarchical features of the Raman spectra are sequentially extracted. The second branch uses the initial mixed features and the hierarchical features of the near-infrared images and the hierarchical features of the Raman spectra extracted in the same layer as inputs. The second branch is sequentially connected with a multi-modal spectral mixing attention model. The multi-modal spectral mixing attention model sequentially outputs mixed hierarchical features. The mixed hierarchical features obtained in the previous layer, the hierarchical features of the near-infrared images and the hierarchical features of the Raman spectra extracted in the next layer are used as the input of the multi-modal spectral mixing attention model in the next layer of the second branch. The third branch is also connected with a multi-modal spectral interaction fusion model after each multi-head self-attention layer, and uses the mixed hierarchical features and the hierarchical features of the Raman spectra in the same layer as inputs. The multi-modal spectral interaction fusion model sequentially outputs Raman spectrum fusion features. The Raman spectrum fusion features obtained in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the third branch. The multi-modal spectral interaction fusion model at the last layer of the third branch is connected to an output layer, and the monitoring index of cell culture predicted by the Raman spectrum fusion features is finally output through the third branch.

[0027] The present invention also discloses a computer terminal, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-mentioned cell culture monitoring method based on the fusion of near-infrared images and Raman spectra of the present invention.

[0028] The cell culture monitoring method based on the fusion of near-infrared images and Raman spectra proposed by the present invention is different from the conventional method of monitoring the cell culture process using a single Raman spectrum. By fusing the local information of the single-point Raman spectrum and the global information of the near-infrared image, the present invention predicts the necessary monitoring indicators (such as cell viability, lactate concentration, ammonium nitrogen concentration, etc.) during the cell culture process. Aiming at the mismatch between the near-infrared image data and the Raman spectrum data in terms of dimension and modality, the present invention designs a multi-level progressive fusion framework with a three-branch structure, and combines a multi-modal spectral hybrid attention model and a multi-modal spectral interaction fusion model to fuse the near-infrared image data and the Raman spectrum data.

[0029] The present invention has the following beneficial effects by adopting the above technical solutions:

[0030] (1) By utilizing the sensitivity and strong penetration ability of near-infrared imaging to polar groups such as water, sugars, and lipids, and the characteristics of large-area component distribution detection, the present invention simultaneously collects near-infrared image data and Raman spectrum data during the cell culture process, and combines a multi-level progressive fusion framework with a three-branch structure to perform data fusion on the near-infrared image data and the Raman spectrum data, providing sufficient buffering for the fusion of multi-modal features, gradually reducing the differences in distribution and expression between different modalities, and alleviating problems such as fusion failure and information bias caused by strong modal heterogeneity in the traditional direct fusion method. Thus, the robustness and expressiveness of the fusion features of the near-infrared data and the Raman spectrum data are improved, and clearer predicted biological indicators during the cell culture process are obtained, enabling the rapid capture of the spatial distribution of components in the culture medium at a macroscopic scale and being suitable for obtaining changes in the overall properties of the sample.

[0031] (2) By introducing a multi-modal spectral hybrid attention model, the hybrid attention mechanism can dynamically adjust the modal weights, increase the attention to another modality when the signal of a certain modality is not obvious, and highlight the key features that are more sensitive to abnormal metabolic changes (such as sudden lactate accumulation or glutamine depletion). And model the semantic dependence relationship between the near-infrared image data and the Raman spectrum data globally, dynamically adjusting the attention weights of different modal information, so as to effectively extract the key features that are more meaningful for discriminating the target metabolic components during the cell culture process.

[0032] (3) The design of the present invention also adopts a multi-modal spectral interaction fusion model to enhance the complementarity between different information sources of near-infrared image data and Raman spectral data, and fully utilizes the differences and complementary characteristics of the two modalities in the spectral response mechanism. The near-infrared image data mainly reflects the broadband absorption characteristics based on intermolecular vibrations, and is good at capturing the overall change trends of macromolecular components such as water, protein, and lipid, and can characterize the dynamic changes of cell density, biomass growth, and nutrients; while the Raman spectral data provides high-resolution structural information based on intramolecular vibration modes, and is particularly sensitive to the types of chemical bonds, functional group configurations, and molecular backbone characteristics, and is more suitable for revealing the structural changes and concentration fluctuations of key metabolites such as glucose, glutamine, lactate, and ammonia. This complementarity between modalities performs a simple and efficient feature fusion of near-infrared image features and Raman spectral features, enabling the fusion model to co-sense the changes in the culture state from two levels: macroscopic cell behavior and microscopic metabolic level, thereby achieving a more comprehensive and accurate process monitoring of the cell culture process.

[0033] (4) In the process of processing near-infrared image data and Raman spectral data, the present invention introduces an explicit two-dimensional position encoding mechanism, and combines the position encoding loss for spatial constraint during the training of the model, enabling the model to accurately perceive the spatial correspondence of Raman spectral sampling points in the near-infrared image. During the model training process, the explicit position encoding losses of the near-infrared image feature branch and the Raman spectral feature branch are used respectively, enhancing the model's understanding of the implicit geometric constraints between the spatial structure and the modalities, constructing a more physically reasonable cross-modal fusion mechanism, and improving the accuracy of dynamic monitoring and prediction of the cell culture process.

[0034] In summary, the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra provided by the present invention collects Raman spectral data and near-infrared image data during the cell culture process, realizes the multi-modal spectral feature fusion of the two types of data by deep learning methods, fully utilizes the high-molecular specificity of Raman spectra and the macroscopic spatial distribution advantages of near-infrared imaging, and dynamically predicts the metabolite indicators during the cell culture process, meeting the higher requirements of modern biomanufacturing for the real-time performance, stability, and generalization ability of process analytical technology (PAT).

[0035] The following further describes the present invention in conjunction with the accompanying drawings and specific embodiments. Description of the Drawings

[0036] Figure 1 It is a schematic flow chart of the fusion of near-infrared image data and Raman spectral data for the cell culture monitoring method of the embodiment.

[0037] Figure 2 It is a schematic diagram of the multi-modal spectral hybrid attention model at the first level of the second branch in the embodiment.

[0038] Figure 3 Schematic diagram of the multimode spectral interaction fusion model at the first level of the third branch in the embodiment.

[0039] Figure 4 Block diagram of the cell culture monitoring system based on the fusion of near-infrared images and Raman spectra in the embodiment. Detailed implementation manners

[0040] Refer to Figure 1 , a method for monitoring cell culture based on the fusion of near-infrared images and Raman spectra provided in this embodiment predicts the monitoring indicators of cell culture by collecting near-infrared image data and Raman spectral data of the cell culture environment during the cell culture process. The monitoring indicators here include, but are not limited to, monitoring indicators such as cell viability, lactic acid concentration, and ammonium nitrogen concentration. The specific prediction process is as follows:

[0041] S1. Perform normalization processing and block coding on the near-infrared image data and Raman spectral data, and splice explicit position coding on the block coding.

[0042] Use Z-score normalization to perform normalization processing on the near-infrared image data and Raman spectral data in the spectral dimension. Specifically, the formulas for performing Z-score normalization on the near-infrared image data NI and Raman spectral data RM are ; where is the near-infrared image data NI or Raman spectral data RM to be normalized, is the mean value of the corresponding data in the corresponding spectral channel dimension, is the variance of the corresponding data in the corresponding spectral channel dimension.

[0043] Perform block coding on the normalized near-infrared image data and Raman spectral data to obtain near-infrared image coding and Raman spectral coding, and add explicit position coding. For the near-infrared image data and Raman spectral data with different data dimensions, the near-infrared image coding uses a three-dimensional image block as a Patch, and the Raman spectral coding uses a small segment of adjacent spectral curve values as a Patch. The explicit position coding adopts a two-dimensional coordinate form, encoding the row and column positions of each image block in the near-infrared image data, and the row and column positions of the sampling points of the Raman spectral data corresponding to the image blocks in the near-infrared image.

[0044] Specifically, perform block coding on the normalized near-infrared image data and Raman spectral data respectively using two-dimensional convolution and one-dimensional convolution to obtain near-infrared image coding and Raman spectral coding , where and Indicates the number of Patches, indicating the dimension of the feature. A single Patch in the near-infrared image encoding corresponds to the corresponding image patch, where is the set block length, and is the number of bands of the near-infrared image; a single Patch of Raman spectral data corresponds to adjacent spectral curve values within a small band, with a size of , where is the number of samples of Raman spectral data during sampling, is the number of adjacent spectral curve values for encoding, and is the number of bands of Raman spectral data.

[0045] For the near-infrared image encoding and the Raman spectrum encoding explicit positional encoding is introduced to represent the absolute position of each Patch on the original image. For example, the positional encoding of the Patch in the first row and first column is (1,1), and it is concatenated to the beginning of the corresponding feature to obtain the near-infrared image feature with positional encoding ; the positional encoding of the Raman spectrum is determined by the pixel position on the corresponding near-infrared image. Whichever Patch the pixel belongs to, the position of that Patch is used as the positional encoding to obtain the Raman spectrum feature with positional information ; if there are multiple Raman spectrum sampling points, the coordinate values are determined by their central position, and the central position calculation formula is , where is the central position of n Raman spectrum sampling points, and is the position of the i-th Raman spectrum sampling point, that is, the abscissa and ordinate values of multiple sampling points are averaged respectively to obtain the explicit positional encoding (m,n) of the corresponding Raman spectrum feature.

[0046] S2. Respectively pass the encoded near-infrared image data and Raman spectral data through the multi-head self-attention mechanism to obtain the corresponding near-infrared image initial feature and Raman spectrum initial feature, and fuse the near-infrared image initial feature and Raman spectrum initial feature through the multi-head cross-attention mechanism to obtain the initial mixed feature.

[0047] Specifically, use the multi-head self-attention layer in the original Transformer to process the near-infrared image feature and the Raman spectrum feature encoded in step 1 respectively to obtain the near-infrared image initial feature and the Raman spectrum initial feature , and then use the initial features of the near-infrared image as the key vector Key and the value vector Value, and the initial features of the Raman spectrum as the query vector Quary, and input them into the multi-head cross-attention layer of the original Transformer to output the initial mixed features. . The multi-head self-attention layer and the multi-head cross-attention layer not only include the calculation of the attention mechanism, but also include the feed-forward network and normalization, etc. The calculation of the attention mechanism belongs to the well-known neural network calculation method, and the specific calculation process of the attention mechanism will not be elaborated in this embodiment.

[0048] S3. Perform multi-level progressive feature fusion on the initial features of the near-infrared image, the initial features of the Raman spectrum, and the initial mixed features through three branches.

[0049] The first branch and the third branch respectively extract the hierarchical features of the near-infrared image of the initial features of the near-infrared image and the hierarchical features of the Raman spectrum of the initial features of the Raman spectrum layer by layer through the multi-layer multi-head self-attention mechanism. The hierarchical features of the near-infrared image extracted in the previous layer are used as the input of the multi-head self-attention mechanism in the next layer of the first branch.

[0050] The second branch passes the hierarchical features of the near-infrared image and the hierarchical features of the Raman spectrum extracted in the same layer together with the initial mixed features through the multi-modal spectral mixing attention model. The multi-modal spectral mixing attention model uses an improved multi-head attention mechanism to calculate the dot product attention matrix of the initial mixed features with the hierarchical features of the near-infrared image and the hierarchical features of the Raman spectrum respectively, and then multiplies the attention matrix element by element with the corresponding hierarchical feature matrix of the near-infrared image or the Raman spectrum to obtain the near-infrared mixed feature and the Raman mixed feature respectively. The near-infrared mixed feature and the Raman mixed feature together with the mixed feature are used as the input of the multi-head attention layer, where the mixed feature is used as the query vector Quary, the near-infrared mixed feature is used as the key vector Key, and the Raman mixed feature is used as the value vector Value, and the mixed hierarchical feature is output through the multi-head attention layer. The mixed hierarchical feature obtained in the previous layer and the hierarchical features of the near-infrared image and the Raman spectrum extracted in the next layer are used as the input of the multi-modal spectral mixing attention model in the next layer of the second branch.

[0051] The specific calculation process is as Figure 2 shown. In the multi-modal spectral mixing attention model of the first layer of the second branch, the initial mixed feature and the initial features of the near-infrared image after the second multi-head self-attention mechanism calculation and the initial features of the Raman spectrum are input into the multi-modal spectral mixing attention model. The multi-modal spectral mixing attention model first combines the initial mixed feature with the initial features of the near-infrared image after the second multi-head self-attention mechanism calculation and the initial features of Raman spectroscopy Calculate the attention respectively to obtain the near-infrared mixed features and Raman mixed features , and their calculation formulas are as follows:

[0052] 、

[0053] 。

[0054] Among them is the scaling factor, and T represents matrix transpose.

[0055] Then use the near-infrared mixed features and the Raman mixed features together as the input of the multi-head attention layer, where the initial mixed features serve as the query vector Quary, the near-infrared mixed features serve as the key vector Key, and the Raman mixed features serve as the value vector Value, and its output mixed hierarchical features serve as the input of the next layer of the second branch. The calculation of this process is as follows:

[0056] 、

[0057] 。

[0058] Among them is the scaling factor, is the output obtained by attention calculation, is the layer normalization operation, is the feed-forward network layer in the original Transformer, and T represents matrix transpose.

[0059] The third branch passes the mixed hierarchical features and Raman spectroscopy hierarchical features of the same layer through the multi-modal spectroscopy interaction fusion model. The multi-modal spectroscopy interaction fusion model calculates the dot-product attention of the mixed hierarchical features processed by the linear layer and the Raman spectroscopy hierarchical features, and obtains the cross-correlation attention weight matrix through Softmax. Calculate the Gaussian position attention matrix based on the near-infrared image space according to the position encoding corresponding to the Raman spectroscopy hierarchical features; after the Gaussian position attention matrix is processed by the linear layer, it is multiplied element by element with the cross-correlation attention weight matrix and the input mixed hierarchical features to obtain the Raman spectroscopy fusion features. The Raman spectroscopy fusion features obtained in the previous layer serve as the input of the multi-head self-attention mechanism in the next layer of the third branch.

[0060] Such as Figure 3As shown in the figure, in the multimodal spectral interaction and fusion model of the first layer of the third branch, the mixed hierarchical features output by the multimodal spectral hybrid attention model of the same layer as the second branch , together with the initial Raman spectral features calculated by the third branch through the multi-head self-attention mechanism , are input into the multimodal spectral interaction and fusion model. The multimodal spectral interaction and fusion model first processes the mixed hierarchical features and the initial Raman spectral features calculated by the third branch through the multi-head self-attention mechanism respectively through the linear layer, and then also calculates the cross-correlation attention weight matrix through attention calculation , and its process is as follows:

[0061] .

[0062] Among them represents the linear layer, is the scaling factor.

[0063] Then, according to the explicit position encoding (m, n) of the Raman spectral features, combined with the near-infrared image coordinates, the Gaussian position attention matrix based on the near-infrared image space is calculated , and the calculation formula of the Gaussian position attention is as follows:

[0064] .

[0065] Among them, refers to one of the elements of the Gaussian position attention matrix , and respectively represent the horizontal and vertical coordinates of a certain Patch of the near-infrared image features, represents the natural exponential operation, represents the diffusion degree of the Gaussian kernel, which is an adjustable hyperparameter; the position coordinates of all Patches in the near-infrared features are calculated to obtain the corresponding Gaussian position attention, and the Gaussian position attention matrix is formed .

[0066] Then, the obtained Gaussian position attention matrix is processed by a learnable linear transformation matrix , and then multiplied element by element with the cross-correlation attention weight matrix to obtain the fusion matrix ; multiply the fusion matrix with the mixed hierarchical features to obtain the fusion features, and finally perform a residual connection with the initial Raman spectral features calculated by the third branch through the multi-head self-attention mechanism to output the Raman spectral fusion features , the process is as follows:

[0067] .

[0068] Where represents element-wise multiplication.

[0069] The Raman spectral fusion features output by this layer of the multi-modal spectral interaction fusion model are used as the input of the multi-head self-attention layer of the next layer of the third branch, and new Raman spectral fusion features are output layer by layer.

[0070] S4. After iterating the three branches in step S3 for N layers, the monitoring index prediction results of cell culture are output through the Raman spectral fusion features finally output by the third branch. N is the number of layers of the three branches. The range of N can be determined according to actual engineering experience to select an initial number of layers N (for example, N = 6). Then, during the training process, check the model accuracy and the occupancy of hardware resources; if the hardware resources are insufficient or the accuracy is high, 1 - 3 layers can be appropriately reduced; if the training accuracy is lower than the required accuracy and the hardware resources are sufficient, 1 - 3 layers can be appropriately increased, and repeatedly adjusted until the model meets the actual accuracy requirements. The more layers there are, the greater the requirements for hardware (graphics card), and the training and inference time costs will also increase. Therefore, on the premise of meeting the accuracy requirements, the number of layers should be as small as possible.

[0071] The above multi-modal spectral hybrid attention model and the multi-modal spectral hybrid attention model are trained on the multi-group near-infrared image data and multi-sampling point Raman spectral data collected during the cell culture process according to the above steps. The cell culture monitoring index results analyzed offline based on the near-infrared image data and Raman spectral data collection time periods are used as the labels for the corresponding training data. During the training process of the multi-modal spectral hybrid attention model and the multi-modal spectral hybrid attention model, the final near-infrared image data position encoding regression loss, Raman spectral data position encoding regression loss, and prediction result regression loss are output by step S4, and the sum loss of the model is obtained through weighted summation. During the actual cell culture monitoring process, the trained multi-modal spectral hybrid attention model and the multi-modal spectral hybrid attention model directly predict the monitoring indexes during the cell culture process.

[0072] That is to say, during the model training process, according to the finally obtained near-infrared hierarchical features and Raman spectral fusion features of the first branch and the third branch, the position encoding regression loss of the first two features is calculated from the final near-infrared hierarchical features and the final Raman spectral fusion features to obtain the near-infrared image data position encoding regression loss Position encoding regression loss of near-infrared image data and Raman spectroscopy data . The position encoding regression loss function uses the MSE loss, and the loss calculation formula is as follows:

[0073] .

[0074] Where and represent the horizontal and vertical coordinate values of the initial position encoding of the near-infrared image features and the initial position encoding of the Raman spectroscopy features, and are the horizontal and vertical coordinate values of the position encoding in the finally output near-infrared stratification and Raman spectroscopy features, represents the number of features.

[0075] And according to the finally output Raman spectroscopy fusion features of the third branch, after being processed by the output layer, the prediction results of the cell culture monitoring indicators are obtained , where K is the number of monitoring indicators; the output layer consists of a fully connected layer and an activation function, and its process formula is as follows:

[0076] .

[0077] Where is a learnable linear transformation matrix, , is a bias term, is an activation function.

[0078] Combined with the actual monitoring indicator results of cell culture from offline analysis of the data collection periods of near-infrared image data and Raman spectroscopy data, calculate the prediction regression loss of the prediction results . The loss function also uses the MSE loss, and the calculation method of the prediction result loss is the same as that of the position encoding regression loss. Then, the position encoding regression loss of the near-infrared image data, the position encoding regression loss of the Raman spectroscopy data, and the prediction result regression loss are weighted and combined to obtain the sum loss , which is used for model training. The sum loss is calculated as follows:

[0079] .

[0080] Where is a settable hyperparameter, .

[0081] See Figure 4 ​​​​, this embodiment also discloses a cell culture monitoring system based on the fusion of near-infrared images and Raman spectra, including an acquisition module 100, a data processing module 200, and a data fusion prediction module 300.

[0082] The acquisition module 100 acquires near-infrared image data and Raman spectrum data during cell culture, including an image acquisition device for acquiring near-infrared images and a spectral instrument for Raman spectrum sampling. These are all conventional devices for cell culture detection, and this embodiment will not elaborate on the specific acquisition methods of near-infrared image data and Raman spectrum data here.

[0083] The data processing module 200 performs normalization processing and block coding on the near-infrared image data and Raman spectrum data, splices explicit position coding on the block coding, and respectively obtains corresponding initial near-infrared image features and initial Raman spectrum features of the near-infrared image data and Raman spectrum data through a multi-head attention mechanism, and fuses the initial near-infrared image features and initial Raman spectrum features through a multi-head cross-attention mechanism to obtain initial mixed features.

[0084] The data fusion prediction module 300 adopts a three-branch multi-level structure to perform multi-level progressive feature fusion on the initial near-infrared image features, initial Raman spectrum features, and initial mixed features. Among them, the first branch 310 and the third branch 330 respectively take the initial near-infrared image features and initial Raman spectrum features as inputs, and multi-head self-attention layers are set layer by layer on the first branch 310 and the third branch 330 to extract hierarchical near-infrared image features and hierarchical Raman spectrum features layer by layer. The second branch 320 takes the initial mixed features and the hierarchical near-infrared image features and hierarchical Raman spectrum features extracted at the same layer as inputs. The second branch 320 is connected to a trained multi-modal spectral hybrid attention model layer by layer, and outputs hybrid hierarchical features layer by layer through the multi-modal spectral hybrid attention model. The hybrid hierarchical features obtained in the previous layer and the hierarchical near-infrared image features and hierarchical Raman spectrum features extracted in the next layer are used as inputs to the multi-modal spectral hybrid attention model in the next layer of the second branch 320. The third branch 330 is connected to a trained multi-modal spectral interaction fusion model after each multi-head self-attention layer, takes the hybrid hierarchical features and hierarchical Raman spectrum features at the same layer as inputs, and outputs Raman spectrum fusion features layer by layer through the multi-modal spectral interaction fusion model. The Raman spectrum fusion features obtained in the previous layer are used as inputs to the multi-head self-attention mechanism in the next layer of the third branch. The multi-modal spectral interaction fusion model at the last layer of the third branch 330 is connected to an output layer, and the monitoring index of cell culture predicted by the Raman spectrum fusion features is finally output through the third branch 330.

[0085] This embodiment also discloses a computer terminal, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it can be used to execute the cell culture monitoring method based on the fusion of near-infrared images and Raman spectra in the above-mentioned embodiment of the present invention.

[0086] It should be understood that in the embodiments of the present invention, the so-called processor may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0087] In this article, the orientation or positional relationship indicated by the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", "vertical", "horizontal", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of clearly expressing the technical solution and description, and therefore cannot be understood as a limitation to the present invention.

[0088] In this article, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion. In addition to the elements listed, it may also include other elements not expressly listed.

[0089] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A cell culture monitoring method based on the fusion of near-infrared images and Raman spectra, characterized in that: Collect near-infrared image data and Raman spectrum data during the cell culture process to predict the monitoring indicators of cell culture. The prediction process includes the following steps: S1. Perform normalization processing and block coding on the near-infrared image data and Raman spectrum data, and splice explicit position coding on the block coding; S2. Respectively pass the encoded near-infrared image data and Raman spectrum data through the multi-head self-attention mechanism to obtain the corresponding initial near-infrared image features and initial Raman spectrum features, and then fuse the initial near-infrared image features and initial Raman spectrum features through the multi-head cross-attention mechanism to obtain the initial mixed features; S3. Perform multi-level progressive feature fusion on the initial near-infrared image features, initial Raman spectrum features, and initial mixed features in three branches. Among them, The first branch and the third branch respectively extract the hierarchical near-infrared image features of the initial near-infrared image features and the hierarchical Raman spectrum features of the initial Raman spectrum features layer by layer through the multi-head self-attention mechanism, The second branch passes the hierarchical near-infrared image features and hierarchical Raman spectrum features extracted in the same layer together with the initial mixed features through the multi-modal spectral mixture attention model to obtain the mixed hierarchical features, The third branch passes the mixed hierarchical features and hierarchical Raman spectrum features in the same layer through the multi-modal spectral interaction fusion model to obtain the Raman spectrum fusion features; S4. After iterating the three branches in step S3 according to the number of hierarchical levels, output the prediction result of the monitoring indicators of cell culture through the Raman spectrum fusion features finally output by the third branch.

2. The cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to claim 1, characterized in that: In step S1, Z-score normalization is used to perform normalization processing on the near-infrared image data and Raman spectrum data in the spectral dimension.

3. The cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to claim 2, characterized in that: For the near-infrared image data and Raman spectrum data with different data dimensions, the near-infrared image data encodes a three-dimensional image block at a time, and the Raman spectrum data encodes a segment of adjacent spectral curve values at a time.

4. The method for monitoring cell culture based on the fusion of near-infrared images and Raman spectra according to claim 1, wherein: The explicit position coding adopts a two-dimensional coordinate form, and encodes the row and column positions of each image block in the near-infrared image data, as well as the row and column positions of the sampling points of the Raman spectrum data corresponding to the image blocks on the near-infrared image. During the multi-modal spectral mixture attention model and the training process of the multi-modal spectral mixture attention model in step S3, the position coding regression loss of the final near-infrared image data, the position coding regression loss of the Raman spectrum data, and the prediction result regression loss are output by step S4, and the sum loss of the model is obtained through weighted summation.

5. The cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to claim 1, characterized in that: The multi-modal spectral mixture attention model adopts an improved multi-head attention mechanism, performs multi-head self-attention mechanism calculations on the initial mixed features respectively with the hierarchical near-infrared image features and hierarchical Raman spectrum features to obtain the near-infrared mixed features and Raman mixed features; input the near-infrared mixed features, Raman mixed features, and mixed features together into the multi-head attention layer, and output the mixed hierarchical features.

6. The method for monitoring cell culture based on the fusion of near-infrared images and Raman spectra according to claim 1, wherein: In the multi-modal spectral interaction and fusion module, the dot-product attention calculation is performed on the Raman spectral hierarchical feature and the hybrid hierarchical feature to obtain the cross-correlation attention weight matrix. The position encoding corresponding to the Raman spectral hierarchical feature is used to calculate the Gaussian position attention matrix based on the near-infrared image space. The cross-correlation attention weight matrix and the Gaussian position attention matrix are multiplied element-wise with the input hybrid hierarchical feature, and a residual connection is made with the Raman spectral hierarchy to output the Raman spectral fusion feature.

7. The cell culture monitoring method based on the fusion of near-infrared images and Raman spectra according to claim 6, characterized in that: Both the Raman spectral hierarchical feature and the hybrid hierarchical feature are processed by a linear layer and then subjected to dot-product attention calculation, and the cross-correlation attention weight matrix is obtained through Softmax.

8. The method for monitoring cell culture based on the fusion of near-infrared images and Raman spectra according to claim 1, wherein: In the iteration of step S4, the near-infrared image hierarchical feature extracted in the previous layer is used as the input of the multi-head self-attention mechanism in the next layer of the first branch. The hybrid hierarchical feature obtained in the previous layer, the near-infrared image hierarchical feature and the Raman spectral hierarchical feature extracted in the next layer are used as the input of the multi-modal spectral hybrid attention model in the next layer of the second branch. The Raman spectral fusion feature obtained in the previous layer is used as the input of the multi-head self-attention mechanism in the next layer of the third branch.

9. A cell culture monitoring system based on the fusion of near-infrared images and Raman spectra, characterized in that It includes: A collection module that collects near-infrared image data and Raman spectral data during cell culture; A data processing module that normalizes and block-encodes the near-infrared image data and Raman spectral data, splices explicit position encoding on the block encoding, and passes the encoded near-infrared image data and Raman spectral data through a multi-head self-attention mechanism to obtain the corresponding initial near-infrared image feature and initial Raman spectral feature, and fuses the initial near-infrared image feature and the initial Raman spectral feature through a multi-head cross-attention mechanism to obtain the initial hybrid feature; A data fusion and prediction module that uses a three-branch multi-level structure to perform multi-level progressive feature fusion on the initial near-infrared image feature, the initial Raman spectral feature, and the initial hybrid feature. Among them, The first branch and the third branch take the initial near-infrared image feature and the initial Raman spectral feature as inputs respectively. The multi-head self-attention layer is set layer by layer on the first branch and the third branch to extract the near-infrared image hierarchical feature and the Raman spectral hierarchical feature layer by layer. The second branch takes the initial hybrid feature, the near-infrared image hierarchical feature and the Raman spectral hierarchical feature extracted in the same layer as inputs. The multi-modal spectral hybrid attention model is connected layer by layer on the second branch. The multi-modal spectral hybrid attention model outputs the hybrid hierarchical feature layer by layer. The hybrid hierarchical feature obtained in the previous layer, the near-infrared image hierarchical feature and the Raman spectral hierarchical feature extracted in the next layer are used as the input of the multi-modal spectral hybrid attention model in the next layer of the second branch. The third branch is also connected with a multi-modal spectral interaction and fusion model after each multi-head self-attention layer, taking the hybrid hierarchical feature and the Raman spectral hierarchical feature in the same layer as inputs. The multi-modal spectral interaction and fusion model outputs the Raman spectral fusion feature layer by layer. The Raman spectral fusion feature obtained in the previous layer is used as the input of the multi-head self-attention mechanism in the next layer of the third branch. The multi-modal spectral interaction and fusion model at the last layer of the third branch is connected to the output layer, and the monitoring index of cell culture predicted by the Raman spectral fusion feature is finally output through the third branch.

10. A computer terminal, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1-8.

Images