Bacterial drug resistance prediction method and system based on deep fusion of multi-modal raman spectrum data

Abstract

The application provides a bacterial drug resistance prediction method and system based on multimodal Raman spectrum data deep fusion, comprising: obtaining static Raman spectrum data of target bacteria and target antibiotics and dynamic Raman spectrum data of target bacteria under the action of target antibiotics; embedding coding and feature extraction are respectively performed on the static Raman spectrum data and the dynamic Raman spectrum data to obtain static spectrum features and dynamic spectrum features; the static spectrum features and the dynamic spectrum features are deeply cross-fused to obtain a drug resistance prediction result of the target bacteria by combining a multilayer perception machine. The technical scheme breaks the limitation of long time consumption and single information of traditional drug sensitivity detection, and provides a new drug resistance monitoring scheme which is efficient, accurate and interpretable for clinic.

Description

Technical Field

[0001] This invention belongs to the field of bacterial resistance prediction technology, specifically relating to a method and system for predicting bacterial resistance based on deep fusion of multimodal Raman spectroscopy data. Background Technology

[0002] Traditional bacterial antimicrobial susceptibility testing techniques primarily employ culture methods and phenotypic analysis. These methods require steps such as bacterial isolation, amplification, drug exposure, and phenotypic observation, a process that typically takes 18-24 hours, and for some specific pathogens like Mycobacterium tuberculosis, the detection time can even reach several weeks. This time-consuming nature directly hinders clinicians from rapidly obtaining drug resistance data in the early stages of infection, thus delaying the window of opportunity for precise antibiotic treatment. Furthermore, current technologies can only obtain static resistance phenotypic results and cannot monitor the dynamic responses of bacteria under antibiotic treatment (such as changes in membrane fluidity and metabolic reprogramming), leading to a lag in the assessment of current drug resistance trends.

[0003] In the field of antibiotic resistance mechanism research, while gene sequencing technology can analyze the mutation sites of antibiotic resistance genes, it cannot directly link gene expression with real-time antibiotic resistance phenotypes. Mass spectrometry, while capable of detecting bacterial metabolites, lacks sensitivity to low-abundance molecular responses and struggles to capture dynamic changes. In contrast, Raman spectroscopy, with its advantages of non-invasive detection, high specificity, and molecular fingerprinting capabilities, is gradually demonstrating its potential in antibiotic resistance monitoring. However, static Raman spectroscopy can only acquire static molecular information and cannot capture the real-time dynamic responses of bacteria under antibiotic treatment. Furthermore, existing methods still lack multimodal data fusion capabilities; for example, simple splicing of static and dynamic data cannot effectively uncover higher-order correlations in the spatiotemporal dimensions, and existing feature fusion methods suffer from insufficient versatility in cross-modal correlation modeling.

[0004] In clinical practice, doctors urgently need tools that can monitor the dynamic changes in bacterial resistance in real time to accurately adjust antibiotic dosages and treatment cycles. Currently, clinical practice mainly relies on empirical medication regimens, which can easily lead to undertreatment or overtreatment. Therefore, this invention proposes a bacterial resistance prediction method and system based on deep fusion of multimodal Raman spectroscopy. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method and system for predicting bacterial drug resistance based on deep fusion of multimodal Raman spectroscopy data.

[0006] A method for predicting bacterial resistance based on deep fusion of multimodal Raman spectroscopy data includes:

[0007] Acquire static Raman spectral data of the target bacteria and the target antibiotic, as well as dynamic Raman spectral data of the target bacteria under the action of the target antibiotic;

[0008] Embedding encoding and feature extraction are performed on the static Raman spectral data and the dynamic Raman spectral data respectively to obtain static spectral features and dynamic spectral features;

[0009] The static spectral features and the dynamic spectral features are deeply cross-fused, and combined with a multilayer perceptron to obtain the drug resistance prediction results of the target bacteria.

[0010] Preferably, the method further includes preprocessing the acquired dynamic Raman spectral data, wherein the preprocessing method includes:

[0011] Discrete wavelet transform is used to perform multi-scale analysis on the dynamic Raman spectral data to obtain high-frequency coefficients;

[0012] The high-frequency coefficients are thresholded, and the signal is reconstructed by combining the low-pass filter coefficients and the high-pass filter coefficients to obtain the denoised dynamic Raman spectral data.

[0013] Asymmetric least squares method was used to perform baseline correction on the denoised dynamic Raman spectral data.

[0014] The baseline-corrected dynamic Raman spectral data are subjected to area normalization and standard normal transformation, and the transformed dynamic Raman spectral data are then time-corrected using a dynamic time warping algorithm to obtain the preprocessed dynamic Raman spectral data.

[0015] Preferably, the method for embedding and encoding the static Raman spectral data includes:

[0016] The static Raman spectral data is subjected to matrix multiplication using a preset weight matrix to complete the input embedding of the static Raman spectral data;

[0017] Position encoding is performed on the static Raman spectral data that has been embedded, thus completing the embedding encoding of the static Raman spectral data.

[0018] Preferably, the method for embedding and encoding the preprocessed dynamic Raman spectral data includes:

[0019] The preprocessed dynamic Raman spectral data is discretized according to time series and spatial distribution using the token embedding method to obtain a token embedding representation;

[0020] Add position embedding to the token embedding representation to obtain a token embedding representation with position information, thus completing the embedding encoding of the preprocessed dynamic Raman spectral data.

[0021] Preferably, a Transformer encoder with a preset number of layers is used to extract features from the embedded and encoded static Raman spectral data. The Transformer encoder includes a self-attention mechanism module, a pre-feedback layer, and a second normalization layer. The self-attention mechanism module includes a multi-head attention layer and a first normalization layer. The method for extracting features from the embedded and encoded static Raman spectral data using a Transformer encoder includes:

[0022] A linear transformation is performed on the embedded and encoded static Raman spectral data to obtain a linear matrix;

[0023] A bias in antibiotic-bacterial affinity is introduced into the multi-head attention layer, and the linear matrix is ​​processed to obtain the multi-head attention layer output;

[0024] The output of the multi-head attention layer is normalized in the first layer to obtain the first layer normalized output;

[0025] The output of the self-attention mechanism module is obtained by adding the normalized output of the first layer to the embedded and encoded static Raman spectral data.

[0026] The output of the self-attention mechanism module is input into the pre-feedback layer, and the pre-feedback output is added to the output of the self-attention mechanism module for a second-layer normalization process to obtain the Transformer encoder output.

[0027] The output of the previous Transformer encoder is used as the input of the next Transformer encoder until the output of the last Transformer encoder is obtained, thus obtaining the static spectral features.

[0028] Preferably, a spatiotemporal dual-stream attention mechanism encoder with a preset number of layers is combined with a second multilayer perceptron to extract features from the embedded and encoded dynamic Raman spectral data. The spatiotemporal dual-stream attention mechanism encoder includes a spatiotemporal parallel cross-self-attention mechanism module, a fourth layer of normalization, and a first multilayer perceptron. The spatiotemporal parallel cross-self-attention mechanism module includes a third layer of normalization and a spatiotemporal parallel multi-head attention layer.

[0029] Methods for feature extraction from embedded dynamic Raman spectral data using a spatiotemporal dual-stream attention mechanism encoder combined with a second multilayer perceptron include:

[0030] The embedded and encoded dynamic Raman spectral data is reshaped to obtain spatial and temporal input representations.

[0031] The spatial dimension input representation and the temporal dimension input representation are subjected to a third-level normalization process to obtain a third-level normalized output.

[0032] The normalized output of the third layer is linearly transformed to obtain the spatial dimension correlation matrix and the time dimension correlation matrix.

[0033] The spatial dimension correlation matrix and the temporal dimension correlation matrix are input into the spatiotemporal parallel multi-head attention layer to perform multi-head attention calculation, and the output of the spatiotemporal parallel multi-head attention layer is obtained.

[0034] The output of the spatiotemporal parallel multi-head attention layer is added to the embedded and encoded dynamic Raman spectral data to obtain the output of the spatiotemporal parallel cross self-attention mechanism module;

[0035] The output of the spatiotemporal parallel cross self-attention mechanism module is subjected to fourth-layer normalization processing and first-layer perceptron processing to obtain the first-layer perceptron output.

[0036] The output of the first multilayer perceptron is added to the output of the spatiotemporal parallel cross self-attention mechanism module to obtain the output of the spatiotemporal dual-stream attention mechanism encoder;

[0037] The output of the spatiotemporal dual-stream attention mechanism encoder is processed by a second multilayer perceptron to obtain dynamic spectral features.

[0038] Preferably, the method for deep cross-fusion of the static spectral features and the dynamic spectral features includes:

[0039] The static spectral features and the dynamic spectral features are multiplied by a matrix to obtain the result matrix;

[0040] The sum of squared differences between the resulting matrices is calculated to obtain the spectral feature vector of deep cross-fusion; wherein the sum of squared differences is used to measure the degree of difference between the resulting matrices.

[0041] This invention also provides a bacterial resistance prediction system based on deep fusion of multimodal Raman spectroscopy data, for implementing the method, comprising:

[0042] The data acquisition module is used to acquire static Raman spectral data of the target bacteria and the target antibiotic, as well as dynamic Raman spectral data of the target bacteria under the action of the target antibiotic.

[0043] The feature extraction module is used to perform embedding encoding and feature extraction on the static Raman spectral data and the dynamic Raman spectral data respectively to obtain static spectral features and dynamic spectral features.

[0044] The drug resistance prediction module is used to perform deep cross-fusion of the static spectral features and the dynamic spectral features, and combine them with a multilayer perceptron to obtain the drug resistance prediction results of the target bacteria.

[0045] Compared with existing technologies, the beneficial effects of this invention are as follows: It proposes a method and system for predicting bacterial drug resistance based on deep fusion of multimodal Raman spectroscopy data. This method integrates static Raman spectral molecular information of bacteria / antibiotics, dynamic Raman spectral information from long-term continuous monitoring of single bacteria under antibiotic action, and drug resistance gene information to extract and interact with multimodal deep features, uncovering the correlation between multimodal data, predicting the dynamic development trend of bacterial drug resistance, and shortening the clinical drug resistance detection time. By combining time-resolved CARS microscopy (100 ms / frame temporal resolution) with microfluidic technology, it captures the molecular changes of bacteria under antibiotic action in real time and collects dynamic Raman spectral data. It innovatively uses a spatiotemporal dual-flow attention mechanism to simultaneously mine real-time features of dynamic data from both spatial and temporal dimensions. Simultaneously, it constructs a cross-modal cross-fusion module to effectively integrate interactive dynamic and static spectral features, and combines reinforcement learning and transfer learning to optimize model performance. This technology breaks through the limitations of traditional drug susceptibility testing, which is time-consuming and provides limited information, offering a new, efficient, accurate, and interpretable solution for clinical drug resistance monitoring. Attached Figure Description

[0046] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a bacterial resistance prediction method based on deep fusion of multimodal Raman spectroscopy data, as described in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of uniform frame sampling in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the Transformer model structure used to extract static spectral features of bacteria / antibiotics in an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the self-attention module structure with added bias in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the encoder structure of the spatiotemporal dual-stream attention mechanism according to an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the spatiotemporal parallel multi-head attention layer structure in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] like Figure 1 As shown, the bacterial resistance prediction method based on deep fusion of multimodal Raman spectroscopy data includes:

[0057] S1: Acquire static Raman spectral data of the target bacteria and the target antibiotic, as well as dynamic Raman spectral data of the target bacteria under the action of the target antibiotic. The static Raman spectral data includes both bacterial static Raman spectral data and antibiotic static Raman spectral data.

[0058] In this embodiment, the process of acquiring static Raman spectral data includes:

[0059] (1) Determine the target data

[0060] Bacterial data: Identify the bacterial species to be studied (such as Staphylococcus aureus, Escherichia coli) and their drug resistance phenotypes.

[0061] Antibiotic data: Select antibiotics associated with the target bacteria and determine their characteristic peak range.

[0062] (2) Select database

[0063] Download static Raman spectral data of commonly used clinical bacteria and antibiotics from the following publicly available databases:

[0064] 1) Bacterial Raman spectroscopy data: http: / / www.sers-cbp.net /

[0065] 2) Antibiotic Raman spectroscopy data: http: / / sers.test.bniu.net /

[0066] (3) Data retrieval and filtering

[0067] When searching for target data, you need to enter a specific combination of keywords. Bacterial data must include strain number, culture conditions (such as culture medium type and culture time) and drug resistance information (such as MIC value and drug resistance genotype). Antibiotic data must include molecular structure formula and characteristic peak position.

[0068] The process of acquiring static Raman spectral data includes:

[0069] (1) Data Acquisition

[0070] Dynamic Raman spectroscopy data can monitor the molecular changes of bacteria under the influence of antibiotics in real time. This technique is typically used in conjunction with microfluidic systems and time-resolved Raman microscopy. The experiment employs time-resolved CARS microscopy or confocal Raman microscopy, coupled with a microfluidic chip to precisely control the contact environment between bacteria and antibiotics. Laser wavelengths of 785 nm or 532 nm are usually selected to avoid fluorescence interference, and the power is controlled between 5-20 mW to prevent sample damage. The time resolution is set to 100 ms–1 s / frame to capture rapid dynamic responses, and the flow rate of the microfluidic chip is maintained at 0.5–1 μL / min to ensure uniform exposure of bacteria to a gradient concentration of antibiotics. The specific experimental steps are as follows:

[0071] 1) Sample preparation: Inoculate the target bacteria into the culture channel of the microfluidic chip and maintain their activity using LB medium.

[0072] 2) Antibiotic Introduction: Antibiotics are injected through the side channel of the chip, with the concentration gradient determined based on preliminary experiments. Data Acquisition: Time-series data acquisition is initiated. The system will automatically acquire spectral data at fixed time intervals of 5 minutes, covering the fingerprint area of ​​600-1800 cm⁻¹ and the 2800-3200 cm⁻¹ region. -1 The CH vibration region was continuously monitored for 8 hours, resulting in a spectral dataset of 96 sets of data. The original data were time-series Raman spectra, recorded in the form of multidimensional arrays. Each frame contained spatial coordinates (XY) and a spectral intensity matrix (wavenumber-intensity), stored as a multidimensional array (time × space × wavenumber) or in HDF5 format.

[0073] A further implementation includes preprocessing the acquired dynamic Raman spectral data. Dynamic Raman spectral data records the molecular response characteristics of bacteria under antibiotic treatment; however, due to interference from factors such as instrument noise, environmental disturbances, and the complexity of biological samples, the raw data often contains significant noise and baseline drift. To eliminate the interference from these factors, the system preprocesses the raw spectral data to improve data quality and ensure the accuracy of subsequent analyses.

[0074] Wavelet transform possesses multi-resolution analysis capabilities, effectively separating signals and noise. Therefore, this invention decomposes the signal into different scales by selecting appropriate wavelet basis functions and effectively removes noise distributed in high-frequency detail coefficients through thresholding.

[0075] Preprocessing methods include:

[0076] Discrete wavelet transform (DWT Multi-scale analysis was performed on the dynamic Raman spectral data to obtain high-frequency coefficients; the decomposition process is shown in formulas (1) and (2):

[0077] cA j+1 [n]=∑ k h[k-2n]cA j [k] (1)

[0078] cD j+1 [n]=∑ k g[k-2n]cA j [k] (2)

[0079] Where cA j+1 [n] and cD j+1 [n] represents the approximate coefficients obtained from the (j+1)th layer decomposition, h[k-2n] and h[k-2n] are the low-pass filter coefficients, and cA j [k] is the approximation coefficient of the j-th layer.

[0080] Thresholding is applied to the high-frequency coefficients, and the signal is reconstructed by combining the low-pass filter coefficients and the high-pass filter coefficients to obtain the denoised dynamic Raman spectrum data; specifically, the soft threshold function is shown in formula (3).

[0081]

[0082] Where sign(cD) j [n]) is a sign function used to obtain cD j The symbol [n] represents the threshold value, where T is the set threshold. (cD) j [n]|-T) + Indicates when |cD j When [n]|-T>0, take cD. j [n]|-T, otherwise take 0.

[0083] Finally, the signal is reconstructed. The reconstruction process is shown in Equation (4), where h[n] and g[n] are the low-pass and high-pass filter coefficients, respectively, and cA and cD are the approximation and detail coefficients.

[0084] cA j [n]=∑ k h[n-2k]cA j+1 [k]+∑ k g[n-2k]cD j+1 [k] (4)

[0085] The baseline correction of the denoised dynamic Raman spectral data is performed using the asymmetric least squares method. After signal denoising, in order to eliminate the influence of sample autofluorescence and instrument system errors, the baseline drift problem in the signal needs to be further processed. During dynamic monitoring, the baseline drift manifests as a low-frequency background signal that changes with time, and its intensity is often 1-2 orders of magnitude higher than the Raman characteristic peaks. This drift will seriously interfere with the identification and quantitative analysis of characteristic peaks. Traditional baseline correction methods are not effective in processing dynamic data. Therefore, this invention adopts an improved asymmetric least squares method. The objective function is shown in formula (5), and the weight update is shown in formula (6), where λ controls the smoothness and p controls the asymmetry.

[0086] min b ∑ i w i (y i -b i ) 2 +λ∑ i (Δ 2 b i ) 2 (5)

[0087]

[0088] Where w i It is the weighting coefficient, y i It is the original signal value, b i This is the baseline estimate.

[0089] The dynamic Raman spectral data after baseline correction were normalized by area and subjected to standard normal transformation. The transformed dynamic Raman spectral data were then time-corrected using a dynamic time warping algorithm to obtain preprocessed dynamic Raman spectral data. Specifically, after baseline correction, the spectral intensities collected at different time points may exhibit systematic differences due to factors such as laser power fluctuations and sample concentration changes. Normalization is required to eliminate systematic biases. Area normalization eliminates overall intensity changes by dividing by the total spectral area, retains relative peak intensity information, and studies the changes in the intensity ratio of characteristic peaks. The area normalization formula is shown in (7), where Δx j Wavenumber interval:

[0090]

[0091] The standard normal transformation centers the intensity at each wavenumber point and scales the standard deviation, making it suitable for multi-sample integrated analysis. Here, μ and σ represent the mean and standard deviation of all spectra at the wavenumber, respectively. The standard normal transformation is shown in equation (8):

[0092]

[0093] During the acquisition of dynamic data in microfluidic experiments, unstable flow velocity can lead to a shift in the acquisition time point, requiring time correction to ensure the accuracy of dynamic analysis. DTW (Dynamic Time Warping) aligns the time series by finding the path with the minimum cumulative distance, allowing for nonlinear time scaling and handling dynamic data with unstable flow velocity. The distance matrix is ​​shown in formula (9), where D(i,j) represents the minimum cumulative distance from the starting point to (i,j). This represents the value of the first time series at time i. Let γ(i,j) represent the value of the second time series at time j, where i represents the time index of the first series and j represents the time index of the second series. The cumulative distance is shown in formula (10), where γ(i,j) represents the total distance of the optimal path from the starting point to (i,j), W(i,j) represents the local distance of the current point, and min{} represents the minimum cumulative distance of the previous step.

[0094]

[0095] The preprocessing described above resulted in a time-space spectral dataset with a high signal-to-noise ratio and stable baseline. This process effectively eliminated instrument errors and environmental interference, significantly improved data quality, and allowed weak Raman characteristic peaks to be clearly presented.

[0096] This embodiment also collected data on drug resistance and antibiotic-bacterial affinity:

[0097] Obtaining drug resistance data:

[0098] CARD (https: / / card.mcmaster.ca / ) is a comprehensive database of antibiotic resistance genes, containing: sequences, functions, and classifications of resistance genes (AROs), resistance mechanisms (such as enzyme degradation and efflux pumps), and association data with clinical phenotypes (such as MIC values). CARD provides various data files, with the core data package containing all resistance genes, protein sequences, and Ontology (AROs). The file content includes resistance gene ontology (structured classification), resistance protein sequences (FASTA format), resistance gene nucleic acid sequences, and prediction rules for resistance genes.

[0099] (1) Search methods

[0100] Keyword search (suitable for rapid location): Enter gene name, antibiotic, or species. For example, searching for blaCTX-M will retrieve annotations for all CTX-M type β-lactamase genes.

[0101] Advanced Filtering (for batch retrieval): In the "Advanced Search" module, select filtering criteria such as genes, drug resistance mechanisms, and associated pathogens to filter out the data you need.

[0102] (2) Data Download

[0103] Download a single record: Click on the target gene on the search results page, and then download the following information after entering the details page: FASTA sequence (gene / protein) and annotation information (JSON / TSV format, including drug resistance rules and references).

[0104] Batch download: Obtain the complete dataset via the "Download" page.

[0105] (3) Filter valid data:

[0106] Remove duplicate or incompletely annotated records, such as those lacking a description of drug resistance mechanisms.

[0107] Genes with experimental validation are retained and labeled "Experimental Evidence" in CARD.

[0108] Obtaining antibiotic-bacterial affinity data:

[0109] The PubChem database (https: / / pubchem.ncbi.nlm.nih.gov / ) contains over 110 million compounds (including antibiotics, small molecule drugs, etc.) and provides bioactivity data (such as binding affinity, inhibition constant, MIC value, etc.), including compound-target interaction information (such as data on the binding of antibiotics to bacterial proteins).

[0110] Access the PubChem BioAssay database, enter keywords such as antibiotic name or bacterial name in the search bar, or use advanced search functions to limit the search to conditions such as experiment type and species. Filter for biochemical experimental data related to antibiotic-bacterial interactions and check for experimental results that directly measure affinity or related activities. After finding entries related to the target antibiotic or bacteria, view the "Related Data" section and use literature citations, patent information, etc., to trace related research and obtain affinity data clues.

[0111] S2: Embedding encoding and feature extraction are performed on both static and dynamic Raman spectral data to obtain static and dynamic spectral features. Addressing the limitation of existing technologies in real-time monitoring of dynamic changes in drug resistance, this model predicts the dynamic development trend of bacterial drug resistance through the fusion of multimodal data, significantly shortening the clinical drug resistance detection time and thus assisting doctors in accurately determining drug dosage and duration.

[0112] However, feature extraction from dynamic data presents numerous challenges. In terms of spatiotemporal characteristics, the molecular changes in bacteria under antibiotic treatment are continuous and rapid over time, while spatially they vary due to differences in different intracellular regions. Therefore, accurately capturing key features within these spatiotemporal changes is a crucial problem that urgently needs to be solved. Furthermore, dynamic data features are highly heterogeneous; the dynamic response patterns produced by different bacterial species and different antibiotics vary greatly, making it difficult to use unified standards and methods for feature extraction and analysis. This further increases the difficulty of constructing a universal feature extraction model.

[0113] Traditional Transformers struggle to effectively extract both temporal and spatial features simultaneously. Their self-attention mechanism primarily captures sequence dependencies without explicitly distinguishing between temporal and spatial dimensions. When processing dynamic Raman spectroscopy data for bacterial resistance monitoring, the complex spatiotemporal variations of this data make traditional Transformers ineffective. Furthermore, instrument noise, sample complexity, environmental interference, and the high heterogeneity of data features all hinder the accurate extraction of effective spatiotemporal features by traditional Transformers. They also cannot employ a unified method to handle the diverse and complex spatiotemporal characteristics, thus preventing the efficient simultaneous extraction of both temporal and spatial features.

[0114] A further implementation method for embedding and encoding static Raman spectral data includes:

[0115] The static Raman spectral data is embedded by performing matrix multiplication on the static Raman spectral data using a preset weight matrix. Specifically, the original Raman spectral data is multiplied using a learnable weight matrix. The original Raman spectral data is defined as x with dimension n0, and mapped to a low-dimensional space y with dimension m0. The weight matrix W0 is defined with dimension m0×n0. The operation y = W0x yields the low-dimensional vector y. This process effectively reduces the data dimensionality while preserving key features.

[0116] Position encoding is performed on the embedded static Raman spectral data to complete the embedding encoding of the static Raman spectral data. Specifically, position encoding information is performed on the embedded data. For a dt-dimensional vector y, its i0th element... The location code calculation method is shown in formulas (11) and (12):

[0117]

[0118]

[0119] Where pos represents the position of the data in the sequence, i0 represents the index (dimension) of the vector element, and dt represents the dimension of the embedding vector. In this way, positional encoding information is added to each data point, resulting in the final embedded and encoded static data, which is used in subsequent feature extraction stages.

[0120] A further implementation method for embedding and encoding the preprocessed dynamic Raman spectral data includes:

[0121] The preprocessed dynamic Raman spectral data is discretized according to its time series and spatial distribution using a token embedding method to obtain a token embedding representation. In this embodiment, considering the changing characteristics of dynamic data in both time and space, the dynamic Raman spectral data is discretized according to its time series and spatial distribution using a token embedding method, transforming it into a token sequence suitable for model processing. For example, bacterial Raman spectral data at different time points are divided into multiple time slices, and the data within each time slice is further spatially segmented, with each small block of data then converted into a token.

[0122] Therefore, this invention divides the dynamic data collected for each bacterial strain at 5 mins / time intervals into nt = 96 frames, i.e., nt images, in chronological order. Each image frame is spatially divided into non-overlapping nw blocks, i.e., x n1 …x nw Each block can be considered a token. All tokens from nt images are concatenated sequentially to form the input sequence, containing a total of N = nt × nw tokens. The feature dimension is set to d, and the final token embedding representation is defined as... like Figure 2 As shown.

[0123] Adding position embedding to the token embedding representation yields a token embedding representation with positional information, completing the embedding encoding of the preprocessed dynamic Raman spectral data. Specifically, position embedding is then added to these N non-overlapping image blocks, with the encoding method being the same as the Transformer's positional encoding method, as shown in formulas (11) and (12). After token embedding and position embedding, a token embedding representation with positional information is obtained, defined as z, as shown in formula (13). z is derived from the learnable embedding vector z cls and the embedding vector E of each image patch k It consists of, etc., plus position code p0. Where z cls For classifying tokens, the projection of E is equivalent to a 2D convolution operation. To preserve positional information, a learnable positional embedding p0∈R is added to z. N×dEach token is assigned its position information in the time and space sequence so that the model can perceive the order and positional relationship of the data.

[0124] z = [z cls ,Ex1,Ex2,…,Ex N ]+p0 (13)

[0125] In the process of building a high-precision bacterial resistance prediction model, feature extraction is a crucial core step. Its accuracy and efficiency play a decisive role in the overall performance of the model. Feature extraction of static data and dynamic data each have unique processes and significance.

[0126] First, the embedded and encoded bacterial Raman data and antibiotic Raman data are input into separate Transformer modules. After processing by an S-layer Transformer mechanism, the features of the bacterial and antibiotic Raman data are output, respectively. The specific value of S is learned through model training. Figure 3 As shown, each Transformer layer primarily consists of a self-attention mechanism, which extracts features from the Raman spectral data of bacteria and antibiotics, uncovering latent features and patterns within the data. Notably, this invention adds a bias related to antibiotic-bacterial affinity to the multi-head attention layer on top of the basic self-attention mechanism, enabling the model to achieve local attention regarding the correlation between antibiotics and bacteria, such as... Figure 4 As shown, the bias is introduced into the scaled dot product attention mechanism layer of the self-attention module.

[0127] A further implementation involves using a Transformer encoder with a preset number of layers (S layers) to extract features from the embedded static Raman spectral data. The Transformer encoder includes a self-attention mechanism module, a pre-feedback layer, and a second normalization layer. The self-attention mechanism module includes a multi-head attention layer and a first normalization layer. The processing procedures for bacteria and antibiotics are the same; taking bacteria as an example, the method for extracting features from the embedded static Raman spectral data using a Transformer encoder includes:

[0128] For the embedded encoded static Raman spectral data X B Perform a linear transformation to obtain a linear matrix (query) Q. B (key)K B and (value)V B And Q B =K B =V B .

[0129] A bias in antibiotic-bacterial affinity is introduced into the multi-head attention layer to process the linear matrix and obtain the multi-head attention layer output; specifically, these three matrices are input into the multi-head attention layer respectively, and Q is processed. B and K B Perform the dot product calculation and divide the result by Add the deviation value, and then use softmax to obtain V. B The weights are then used to obtain the output of the multi-head attention layer, Attention(Q). B ,K B V B This can be represented as:

[0130]

[0131] Where, d k It is Q B K B and V B The bias is a deviation value calculated based on antibiotic-bacterial affinity data.

[0132] Each attention mechanism consists of h parallel-running multi-head attention layers, 3h+1 linear layers, and one connection operation, where h is the number of heads, which can be learned. Define the h-th... i The output of the multi-head attention layer is head b If b = 1, 2, ..., h, then head a This can be expressed as:

[0133]

[0134] Here, the input Q = K = V of the linear layer is the input X of the Transformer module. B . It is a linear projection matrix.

[0135] Finally, the outputs of the multi-head attention layer are concatenated and passed to the linear layer to obtain the output of the attention mechanism layer. MultiHead(Q,K,V) can be expressed as...

[0136] MultiHead(Q,K,V)=Concat(head1,...,head h W O (16)

[0137] Among them, W O It is a linear projection matrix.

[0138] The output of the multi-head attention layer is normalized in the first layer to obtain the first layer normalized output;

[0139] The output of the first-layer normalized output is added to the embedded and encoded static Raman spectral data to obtain the output of the self-attention mechanism module; specifically, the final output Y is obtained. B , can be represented as:

[0140] Y B =LN(MultiHead(Q,K,V))+X B (17)

[0141] LN() represents the layer normalization operation.

[0142] The output of the self-attention mechanism module is input into the pre-feedback layer, and the pre-feedback output is added to the output of the self-attention mechanism module for the second layer of normalization to obtain the Transformer encoder output.

[0143] The output of the previous Transformer encoder is used as the input of the next Transformer encoder, and so on, until the output of the last Transformer encoder is obtained, thus acquiring static spectral features. Specifically, Y... B The input is fed into the pre-feedback mechanism, added to itself, and then subjected to a layer normalization operation to obtain the final output, which can be represented as:

[0144] Y outB =LN(Y B +FeedForword(Y B (18)

[0145] Among them, Y outB The output of the first-layer Transformer module is then used as input to the second-layer Transformer module, and so on, until the output of the S-layer Transformer module is obtained. This refers to the feature extraction results obtained from bacterial processing using a Transformer encoder. These will subsequently be fed into a cross-fusion module for deep interaction with the dynamic data feature extraction results and the antibiotic static data feature extraction results. The antibiotic static data feature extraction process is the same as that of bacterial static data feature extraction, and will not be repeated here. The feature extraction results are represented as follows: Where S is the number of layers in the Transformer encoder used to extract features from static antibiotic data, which can be obtained through learning.

[0146] The embedded and encoded dynamic data is input into the spatiotemporal dual-stream attention mechanism. The spatiotemporal dual-stream attention encoder also uses self-attention to perform comprehensive feature extraction on the data in both temporal and spatial dimensions. The self-attention mechanism automatically calculates the association weight between each token and all other tokens, thereby capturing the complex dependencies of the data in the spatiotemporal dimensions and successfully extracting high-level features of the dynamic data.

[0147] A further implementation method involves using a spatiotemporal dual-stream attention mechanism encoder with a preset number of layers in combination with a second multilayer perceptron to extract features from the embedded and encoded dynamic Raman spectral data. The spatiotemporal dual-stream attention mechanism encoder includes a spatiotemporal parallel cross-self-attention mechanism module, a fourth layer of normalization, and a first multilayer perceptron. The spatiotemporal parallel cross-self-attention mechanism module includes a third layer of normalization and a spatiotemporal parallel multi-head attention layer.

[0148] Methods for feature extraction from embedded dynamic Raman spectral data using a spatiotemporal dual-stream attention mechanism encoder combined with a second multilayer perceptron include:

[0149] The embedded and encoded dynamic Raman spectral data (dynamic Raman matrix z) is reshaped to obtain spatial and temporal input representations. Specifically, the dynamic Raman matrix z, after token embedding and position embedding, is input into the spatiotemporal dual-stream attention module. After processing by an L-layer spatiotemporal dual-stream attention mechanism encoder, the bacterial dynamic Raman spectral feature extraction result is output. The specific value of L is learned after model training. Each layer of the spatiotemporal dual-stream attention mechanism encoder consists of a spatiotemporal parallel cross-self-attention mechanism, layer normalization (LN), and a multilayer perceptron (MLP), such as... Figure 5 As shown.

[0150] The spatial and temporal input representations are subjected to a third-level normalization process to obtain the third-level normalized output. Then, the input of the l-th (l = 1, 2, ..., L) layer is defined as z. l The input z of the first layer 1 =z. (The last part, "z", appears to be a typo and can be left as is.) l Reshaping is performed to obtain input representations in both spatial and temporal dimensions. and Then perform layer normalization, and then convert the results... and The data is fed into a spatiotemporal parallel multi-head attention layer for parallel multi-head self-attention operations in both the temporal and spatial dimensions, such as... Figure 6 As shown, the spatiotemporal parallel multi-head attention layer includes a spatial head and a temporal head, both of which include a scaling dot product attention mechanism layer.

[0151] The normalized output of the third layer is linearly transformed to obtain the spatial and temporal correlation matrices. Here, the multi-head attention operations in the spatial and temporal dimensions are represented by MSA(), which is similar to the multi-head attention mechanism of Transformer. First, the spatial and temporal correlation matrices are... and The result is linearly transformed to obtain the spatial dimension correlation matrix Q. s K s and V s and the time-dimensional correlation matrix Q t K t and V t， Obtain the label Y for the spatial dimension. s and the label Y for the time dimension t Among them, d k1 It is Q s K s and V s The dimension, d k2 It is Q t K t and V t Dimensions.

[0152]

[0153] The spatial and temporal correlation matrices are input into the spatiotemporal parallel multi-head attention layer for multi-head attention computation, yielding the output of the spatiotemporal parallel multi-head attention layer. Specifically, the spatial and temporal multi-head attention layers are respectively composed of h... s and h t It consists of multiple parallel-running multi-head attention layers, where h s and h t The number of heads can be learned. The multi-head attention mechanism in the spatial and temporal dimensions is similar to the multi-head attention mechanism of the Transformer, and can be found in formulas (15) and (16), based on Y. s and Y t The outputs of the multi-head attention module in the computational spatial and temporal dimensions are represented as follows: and After concatenating and linearizing the results, the output Y of the spatiotemporal parallel multi-head attention module is finally obtained. l That is, the output of the MSA() operation, the calculation process is shown in formula (21), where W b The weights are linear.

[0154]

[0155] The output of the spatiotemporal parallel multi-head attention layer is added to the embedded and encoded dynamic Raman spectral data to obtain the output of the spatiotemporal parallel cross-self-attention mechanism module; specifically, the output of the spatiotemporal parallel cross-self-attention module is defined as y. l The calculation method is shown in formula (22).

[0156]

[0157] The output of the spatiotemporal parallel cross-attention mechanism module is subjected to fourth-layer normalization and first-layer perceptron processing to obtain the first-layer perceptron output; next, the output y is processed... l Layer normalization (LN()) and multilayer perceptron (MLP()) are performed sequentially to obtain the output of the l-th layer spatiotemporal dual-stream attention mechanism encoder, which is also the input of the next layer spatiotemporal dual-stream attention mechanism encoder, defined as z. l+1 The calculation is shown in formula (23).

[0158] z l+1 =MLP(LN(y l ))+y l (twenty three)

[0159] The output of the first multilayer perceptron is added to the output of the spatiotemporal parallel cross self-attention mechanism module to obtain the output of the spatiotemporal dual-stream attention mechanism encoder;

[0160] The output of the spatiotemporal dual-stream attention mechanism encoder is processed by a second multilayer perceptron to obtain dynamic spectral features. Specifically, z l+1 The input is fed into the next layer (the (l+1)th layer) of the spatiotemporal dual-stream attention mechanism module, and so on, until the output of the Lth layer spatiotemporal dual-stream attention mechanism module is obtained. The output of the spatiotemporal dual-stream attention mechanism encoder is then processed by a multilayer perceptron to obtain the feature extraction result of the dynamic spectral data, denoted as Y. outD This data will then be fed into the cross-fusion module as input, where it will interact deeply with the feature extraction results of bacterial and antibiotic static data. This innovative spatiotemporal parallel cross-self-attention strategy retains the global modeling capability of traditional self-attention mechanisms while significantly improving the model's accuracy in analyzing microbial metabolic dynamics through feature decoupling in the spatial-temporal dimensions.

[0161] S3: Deeply cross-fuse static and dynamic spectral features, and combine them with a multilayer perceptron to obtain the drug resistance prediction results of the target bacteria.

[0162] A further implementation method involves deep cross-fusion of static and dynamic spectral features, including:

[0163] The static spectral features and dynamic spectral features are multiplied by a matrix to obtain the result matrix;

[0164] The sum of squared differences between the resulting matrices is calculated to obtain the spectral feature vector of deep cross-fusion; where the sum of squared differences is used to measure the degree of difference between the resulting matrices.

[0165] In the field of bacterial resistance prediction research, different features originate from different information sources. Integrating these features can construct a comprehensive multimodal description, enabling a more accurate modeling process. To achieve accurate prediction of bacterial resistance, dynamic data features extracted by a spatiotemporal dual-stream attention encoder and static bacterial and antibiotic data features extracted by a Transformer need to be fed into a cross-fusion module for deep cross-fusion. Traditional cross-fusion methods, such as feature concatenation, simply connect features from different sources in sequence without considering the intrinsic correlation and importance differences between features, which can mask key features and affect the accuracy of bacterial resistance prediction. While weighted summation considers certain weights, these weights are usually pre-set or simply learned, making it difficult to accurately capture the true relationships between different feature types in complex scenarios.

[0166] In clinical testing, bacterial resistance analysis faces the challenge of integrating multi-source, dynamic, and complex features. This is particularly true when the sample contains multiple pathogens with overlapping Raman spectral characteristics; traditional methods often struggle to effectively distinguish and fuse this information, limiting model performance. Therefore, to achieve accurate prediction of bacterial resistance, this invention employs a novel cross-fusion method.

[0167] The dynamic feature extraction result obtained by the encoder using the spatiotemporal dual-stream attention mechanism is represented as H. d =Y outD The static feature extraction results of bacteria and antibiotics using Transformer are represented as H, respectively. b and H a , and The matrix multiplication of these three feature vectors yields a matrix used for subsequent calculation of the loss function, which assists in model optimization, as shown in formulas (24), (25), and (26).

[0168] r1 = H d T H b (twenty four)

[0169] r2=H d T H a (25)

[0170] r3=H a TH b (26)

[0171] Next, the formula for the sum of squares of the differences is used to measure the degree of difference between the two sets of values ​​by calculating the sum of the squares of their differences. The smaller the difference, the smaller the loss value and the better the fusion effect, as shown in formulas (27) and (28):

[0172] L1 = ∑(r1 - r3) 2 (27)

[0173] L2 = ∑(r2 - r3) 2 (28)

[0174] L1 and L2 reflect the degree of difference between these eigenvectors. They integrate the information of two eigenvectors into a scalar value, which measures the similarity or difference between the two eigenvectors. To a certain extent, they fuse their feature information and prepare for subsequent analysis.

[0175] Then, the results L1 and L2 obtained from the cross-fusion and loss calculation processes are used as inputs and fed into a multilayer perceptron (MLP). The MLP performs multiple linear and nonlinear transformations to further extract key information from the features and uncover more complex higher-order relationships between features. Its output is denoted as h. out Compared to the previous feature fusion stage, the feature representation after MLP processing is more abstract and refined, effectively removing redundant information and highlighting feature dimensions that are more valuable to model decision-making.

[0176] Finally, the feature output h obtained after MLP processing is... out This is combined with the parameter weights determined from the previously downloaded drug resistance data. Let the weight matrix obtained after preprocessing the downloaded drug resistance data be... (n1 is the number of drug resistance categories, m1 is the dimension of the MLP output feature vector), the bias vector is b r ∈R n The feature vector h output by the MLP out With weight matrix W r Perform matrix multiplication and add the bias vector b. r The predicted score vector s0 is obtained as shown in formula (29):

[0177] s0 = W r h out +b r (29)

[0178] The predicted score vector s is then processed by the Softmax function to convert it into a probability distribution, as shown in formula (30):

[0179]

[0180] Each element in p0 This indicates that bacteria exhibit nth response to a specific antibiotic. c The probability of a certain antibiotic resistance state is used to determine antibiotic resistance. A probability threshold τ is set. If max(p0) > τ, the bacteria are determined to be resistant to the antibiotic, and the category corresponding to the maximum value is the specific resistance type. If max(p0) ≤ τ, the bacteria are determined to be sensitive to the antibiotic.

[0181] This embodiment provides a specific implementation process for model training and optimization:

[0182] (1) Dataset partitioning

[0183] To ensure effective training and accurate evaluation of the model, this paper scientifically divides the dynamic dataset:

[0184] Training set (5 sets, 0-8 hours of data): contains complete dynamic Raman spectral data (96 frames, acquired every 5 minutes), covering the complete dynamic response of bacteria under the action of antibiotics.

[0185] Test set (1 set, 0-2 hours data + prediction for the next 6 hours): Only the data for the first 2 hours is provided, and the model is required to predict the trend of drug resistance development in the next 6 hours.

[0186] This classification reflects the need in clinical settings to predict dynamic changes in bacterial resistance based on early data.

[0187] (2) Training and Testing

[0188] First, the model was trained end-to-end using the training set (0-8 hours of data) to optimize model parameters. The loss function combined mean squared error (MSE) and cross-entropy to optimize dynamic feature fitting and drug resistance classification tasks, respectively. Next, the model was input with the data from the first 2 hours of the test set, outputting predictions for the next 6 hours (such as drug resistance probability and trends in key metabolic peaks). Finally, the predicted trends for the next 6 hours were compared with the actual dynamic data from hours 2-8 of the test set.

[0189] (3) Error Analysis and Optimization

[0190] Mean squared error (MSE) is used to measure the average squared error between the predicted and actual values. MSE can be expressed as:

[0191]

[0192] Where n 0It is the number of test set samples. It is the pth f The predicted value for each sample, It is the pth f The true value of each sample.

[0193] In addition, the mean absolute error (MAE) can be calculated to more intuitively reflect the average absolute deviation between the predicted and actual values. MAE can be expressed as:

[0194]

[0195] Finally, a reasonable error threshold is set based on clinical needs. This threshold is used to determine the acceptability of the model's prediction results. For example, if the error in predicting bacterial resistance needs to be controlled within a certain range in clinical applications to ensure the accuracy of medication decisions, then an appropriate MSE or MAE threshold can be set based on this requirement. Next, the calculated error index is compared with the set error threshold. If the error on the test set is greater than the threshold, it indicates that the model performs poorly in predicting the test set data and requires further training and optimization.

[0196] Through the above "training-evaluation-optimization" cycle, the performance of the model is continuously improved until the error of the prediction rate is less than the set threshold, meeting the clinical requirements, and finally used for actual bacterial resistance detection.

[0197] Example 2

[0198] This invention also provides a bacterial resistance prediction system based on deep fusion of multimodal Raman spectroscopy data, and a method for implementing this system, including:

[0199] The data acquisition module is used to acquire static Raman spectral data of the target bacteria and the target antibiotic, as well as dynamic Raman spectral data of the target bacteria under the action of the target antibiotic.

[0200] The feature extraction module is used to perform embedding encoding and feature extraction on static Raman spectral data and dynamic Raman spectral data respectively to obtain static spectral features and dynamic spectral features.

[0201] The drug resistance prediction module is used to perform deep cross-fusion of static and dynamic spectral features, and combine them with a multilayer perceptron to obtain the drug resistance prediction results of the target bacteria.

[0202] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for predicting bacterial drug resistance based on deep fusion of multimodal Raman spectroscopy data, characterized in that, include: Acquire static Raman spectral data of the target bacteria and the target antibiotic, as well as dynamic Raman spectral data of the target bacteria under the action of the target antibiotic; Embedding encoding and feature extraction are performed on the static Raman spectral data and the dynamic Raman spectral data respectively to obtain static spectral features and dynamic spectral features; The static spectral features and the dynamic spectral features are deeply cross-fused, and combined with a multilayer perceptron to obtain the drug resistance prediction results of the target bacteria. A Transformer encoder with a preset number of layers is used to extract features from the embedded and encoded static Raman spectral data. The Transformer encoder includes a self-attention mechanism module, a pre-feedback layer, and a second normalization layer. The self-attention mechanism module includes a multi-head attention layer and a first normalization layer. The method for extracting features from the embedded and encoded static Raman spectral data using the Transformer encoder includes: A linear transformation is performed on the embedded and encoded static Raman spectral data to obtain a linear matrix; A bias in antibiotic-bacterial affinity is introduced into the multi-head attention layer, and the linear matrix is ​​processed to obtain the multi-head attention layer output; The output of the multi-head attention layer is normalized in the first layer to obtain the first layer normalized output; The output of the self-attention mechanism module is obtained by adding the normalized output of the first layer to the embedded and encoded static Raman spectral data. The output of the self-attention mechanism module is input into the pre-feedback layer, and the pre-feedback output is added to the output of the self-attention mechanism module for a second-layer normalization process to obtain the Transformer encoder output. The output of the previous Transformer encoder is used as the input of the next Transformer encoder until the output of the last Transformer encoder is obtained, thus obtaining the static spectral features.

2. The method according to claim 1, characterized in that, It also includes preprocessing the acquired dynamic Raman spectral data, and the preprocessing methods include: Discrete wavelet transform is used to perform multi-scale analysis on the dynamic Raman spectral data to obtain high-frequency coefficients; The high-frequency coefficients are thresholded, and the signal is reconstructed by combining the low-pass filter coefficients and the high-pass filter coefficients to obtain the denoised dynamic Raman spectral data. Asymmetric least squares method was used to perform baseline correction on the denoised dynamic Raman spectral data. The baseline-corrected dynamic Raman spectral data are subjected to area normalization and standard normal transformation, and the transformed dynamic Raman spectral data are then time-corrected using a dynamic time warping algorithm to obtain the preprocessed dynamic Raman spectral data.

3. The method according to claim 1, characterized in that, The method for embedding and encoding the static Raman spectral data includes: The static Raman spectral data is subjected to matrix multiplication using a preset weight matrix to complete the input embedding of the static Raman spectral data; Position encoding is performed on the static Raman spectral data that has been embedded, thus completing the embedding encoding of the static Raman spectral data.

4. The method according to claim 2, characterized in that, The method for embedding and encoding the preprocessed dynamic Raman spectral data includes: The preprocessed dynamic Raman spectral data is discretized according to time series and spatial distribution using the token embedding method to obtain a token embedding representation; Add position embedding to the token embedding representation to obtain a token embedding representation with position information, thus completing the embedding encoding of the preprocessed dynamic Raman spectral data.

5. The method according to claim 1, characterized in that, A spatiotemporal dual-stream attention mechanism encoder with a preset number of layers is combined with a second multilayer perceptron to extract features from the embedded and encoded dynamic Raman spectral data. The spatiotemporal dual-stream attention mechanism encoder includes a spatiotemporal parallel cross-self-attention mechanism module, a fourth layer normalization, and a first multilayer perceptron; the spatiotemporal parallel cross-self-attention mechanism module includes a third layer normalization and a spatiotemporal parallel multi-head attention layer. Methods for feature extraction from embedded dynamic Raman spectral data using a spatiotemporal dual-stream attention mechanism encoder combined with a second multilayer perceptron include: The embedded and encoded dynamic Raman spectral data is reshaped to obtain spatial and temporal input representations. The spatial dimension input representation and the temporal dimension input representation are subjected to a third-level normalization process to obtain a third-level normalized output. The normalized output of the third layer is linearly transformed to obtain the spatial dimension correlation matrix and the time dimension correlation matrix. The spatial dimension correlation matrix and the temporal dimension correlation matrix are input into the spatiotemporal parallel multi-head attention layer to perform multi-head attention calculation, and the output of the spatiotemporal parallel multi-head attention layer is obtained. The output of the spatiotemporal parallel multi-head attention layer is added to the embedded and encoded dynamic Raman spectral data to obtain the output of the spatiotemporal parallel cross self-attention mechanism module; The output of the spatiotemporal parallel cross self-attention mechanism module is subjected to fourth-layer normalization processing and first-layer perceptron processing to obtain the first-layer perceptron output. The output of the first multilayer perceptron is added to the output of the spatiotemporal parallel cross self-attention mechanism module to obtain the output of the spatiotemporal dual-stream attention mechanism encoder; The output of the spatiotemporal dual-stream attention mechanism encoder is processed by a second multilayer perceptron to obtain dynamic spectral features.

6. The method according to claim 1, characterized in that, The method for deep cross-fusion of the static spectral features and the dynamic spectral features includes: The static spectral features and the dynamic spectral features are multiplied by a matrix to obtain the result matrix; The sum of squared differences between the resulting matrices is calculated to obtain the spectral feature vector of deep cross-fusion; wherein the sum of squared differences is used to measure the degree of difference between the resulting matrices.

7. A bacterial resistance prediction system based on deep fusion of multimodal Raman spectroscopy data, used to implement the method described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire static Raman spectral data of the target bacteria and the target antibiotic, as well as dynamic Raman spectral data of the target bacteria under the action of the target antibiotic. The feature extraction module is used to perform embedding encoding and feature extraction on the static Raman spectral data and the dynamic Raman spectral data respectively to obtain static spectral features and dynamic spectral features. The drug resistance prediction module is used to perform deep cross-fusion of the static spectral features and the dynamic spectral features, and combine them with a multilayer perceptron to obtain the drug resistance prediction results of the target bacteria.

Images