An oral cavity multi-modal data fusion assisted diagnosis method and device based on localization storage and calculation

Abstract

The application discloses a kind of oral cavity multi-modal data fusion auxiliary diagnosis method and equipment based on localization storage and calculation, applied to intelligent medical field.The method comprises the following steps: obtaining the oral cavity full-view digital pathology image of subject, and obtaining M pathology image blocks by block processing;The local pathology visual feature vector is obtained by encoding the pathology image block;Batch transcriptome sequencing data is obtained, and the virtual gene expression vector corresponding to each local pathology visual feature vector space is inferred by spatial mapping;the probability value of lymph node metastasis is output by risk prediction through local pathology visual feature vector and virtual gene expression amount;Wherein, the spatial mapping is local pathology visual vector as anchor point, and the corresponding virtual gene expression vector is obtained by reverse mapping transcript sequencing data to local pathology visual feature vector.

Description

Technical Field

[0001] This invention relates to the field of intelligent healthcare, and more specifically, to a method and device for assisted diagnosis of oral multimodal data fusion based on localized storage computing. Background Technology

[0002] Oral squamous cell carcinoma (OSCC) is one of the most common malignant tumors of the head and neck, characterized by its high invasiveness and tendency to metastasize to lymph nodes. Currently, the "gold standard" for clinical diagnosis of OSCC mainly relies on histopathological examination. With the development of digital pathology technology, physical slides are converted into high-resolution digital slides by scanners, making it possible to use computer-aided diagnostic systems. Current auxiliary analysis technologies are mainly divided into image-based AI diagnosis and independent omics sequencing analysis, but both of these technologies still have significant limitations in practical applications. First, image-based AI-aided diagnostic systems mainly use deep learning algorithms to segment and classify digital slides. These methods are based solely on morphological features and cannot obtain molecular biological information about the tumor interior. Second, traditional molecular omics analysis (such as batch transcriptome sequencing), while capable of identifying biomarkers, suffers from significant "spatial information loss" and "analysis lag." Finally, existing model analyses often rely on cloud servers, involving the external transmission of medical data, posing data security and compliance risks, and increasing processing latency due to data transmission. In summary, the existing technology lacks an auxiliary diagnostic system that can spatially fuse pathological morphology analysis and molecular omics data in a localized environment at low cost and with high speed, and can accurately quantify tumor heterogeneity. Summary of the Invention

[0003] To address the problems existing in at least one of the prior art, the present invention provides a method and device for assisted diagnosis of oral multimodal data fusion based on localized storage computing.

[0004] This invention discloses a method for assisted diagnosis of oral multimodal data fusion based on localized in-store computing, comprising: S101: Acquire the full-field digital pathological image of the subject's oral cavity, divide it into blocks to obtain M pathological image blocks; encode the pathological image blocks to obtain local pathological visual feature vectors; Furthermore, encoding the pathological image block to obtain the local pathological visual feature vector specifically involves: inputting the pathological image block into a neural network for staining normalization processing, and extracting the normalized local pathological visual feature vector.

[0005] Optionally, the neural network includes any one or more of the following: generative adversarial networks, variational autoencoders; Optionally, the neural network is an improved generative adversarial network called StainGAN; the StainGAN network includes a mirror-symmetric generator and a discriminator.

[0006] Optionally, the specific configuration of the StainGAN network architecture is as follows: The generator employs a fully convolutional residual architecture and is configured to receive the original pathological image patch as input. It performs feature extraction and downsampling encoding on the input through multiple convolutional layers, mapping the image into a low-dimensional feature space. Stacked residual modules perform nonlinear coloring style transfer in the deep feature space; finally, the transferred features are upsampled and reconstructed through deconvolution layers to output the coloring-normalized target image patch.

[0007] Optionally, the discriminator employs a Markov discriminator architecture and is configured to receive the target image patch or standard template image patch as input, and map it to... The feature matrix, in which each element corresponds to the true / false discrimination result of a local receptive field patch in the input image patch, forces the generator to retain high-frequency texture details during the generation process.

[0008] S102: Acquire batch transcriptome sequencing data and infer the virtual gene expression vector corresponding to each of the local pathological visual feature vectors through spatial mapping; perform risk prediction and output the probability value of lymph node metastasis through local pathological visual feature vectors and virtual gene expression levels; wherein, the spatial mapping is to use the local pathological visual vector as the anchor point and reverse map the transcription sequencing data to the local pathological visual feature vector to obtain the corresponding virtual gene expression vector.

[0009] Furthermore, the transcriptome sequencing data is a gene expression matrix, and the local pathological visual feature vector includes cell morphology, tissue structure, and tissue texture features in the pathological image. The gene expression matrix is ​​inversely mapped to the cell morphology, tissue structure, and tissue texture features to obtain a virtual gene expression vector with spatial coordinate attributes.

[0010] Optionally, the reverse mapping process is as follows: the gene expression matrix is ​​deconvolutioned to calculate the cell abundance to obtain each cell type; the cell type is matched with the cell morphology, tissue structure, and tissue texture features in the local pathological visual feature vector.

[0011] Alternatively, the matching can be performed using a pre-built prototype library.

[0012] Optionally, the pre-built prototype library construction process is as follows: construct a first prototype vector set from the morphological features in the pathological image, average different morphological features in the first prototype vector set to obtain prototype vectors, each prototype vector representing a cell type, calculate the similarity between each image block of the pathological image and all prototype vectors, determine the cell type of each image block of the pathological image, and obtain the constructed prototype library.

[0013] Optionally, the spatial coordinate attribute is the spatial coordinates of image blocks mapped from cell type to pathological image.

[0014] Optionally, the spatial mapping is a morphology-molecular manifold mapping algorithm; the morphology-molecular manifold mapping algorithm specifically involves: constructing an energy minimization objective function, using the deconvolution result of the cellular components of the batch transcriptome sequencing data as a global macro-constraint, and using the morphological similarity matrix of the local pathological visual feature vectors as a local micro-constraint, and solving for a virtual gene expression vector with spatial coordinate attributes under the mutual constraints of the two.

[0015] Furthermore, the constructed energy minimization objective function includes mutually constraining local microscopic constraints and global macroscopic constraints, and the specific mathematical expression of the energy minimization objective function is as follows: , in, Indicates the problem to be solved, the first... Virtual gene expression vectors for each pathological image patch This represents a batch transcriptome data matrix. This represents the regularization hyperparameter used to balance local micro-constraints and global macro-constraints. Indicates that based on morphological characteristics, it is the first The optimal gene expression prototype vector is dynamically matched to each image patch. The determination process includes: pre-constructing a system containing... A morphological prototype library of cell types was used to extract the first... High-dimensional morphological feature vector of a pathological image patch Calculate the high-dimensional morphological feature vector The average gene expression profile corresponding to the cell type with the highest cosine similarity to the morphological prototype vectors in the prototype library is selected as the mean similarity. .

[0016] Furthermore, the method also includes prediction correction, which uses a virtual gene expression vector as a gating signal to weight and filter the local pathological visual feature vectors to generate a tumor heterogeneity index. The tumor heterogeneity index is used as a correction factor to adjust the risk prediction results, and the probability value of output lymph node metastasis is obtained. Optionally, a cross-gating unit is constructed, using the virtual gene expression vector as the gating signal to perform element-wise weighted filtering of the local pathological visual feature vectors of each pathological image block, in order to filter out morphological noise of no clinical significance; the calculation of the first... Attention weights of individual pathological image patches for predicting lymph node metastasis risk Based on the calculated attention weights For all input local pathological visual feature vectors By performing weighted summation, we obtain the pathological global context vector representing the overall characteristic distribution of the tumor. Calculate the weighted Euclidean distance variance of each local pathological visual feature vector relative to the global pathological context vector, and define it as the tumor heterogeneity index. .

[0017] Preferably, the attention weight The specific calculation formula is as follows: , in, and Indicates the input number of the first... The and the first A local pathological visual feature vector Represents the virtual gene expression vector. and Let represent the learnable projection matrices for visual features and genetic features, respectively. This represents element-wise multiplication. This represents the hyperbolic tangent nonlinear activation function. This represents the transpose of the attention scoring vector. This indicates the total number of high-risk pathological image blocks.

[0018] Optionally, the tumor heterogeneity index The specific calculation formula is as follows: , , in, Represents the global context vector of pathology. The squared L2 norm of a vector. The total number of image patches. Indicators of tumor heterogeneity, when The higher the value, the greater the difference in pathological features within the tumor.

[0019] The present invention also discloses an oral multimodal data fusion-assisted diagnostic system based on localized in-memory computing. The system includes a memory and a processor. The memory stores a computer program, which is executed by the processor to implement the above-mentioned method. The system is logically divided into: Preprocessing module 201: used to acquire the full-field digital pathological image of the subject's oral cavity, divide it into blocks to obtain M pathological image blocks; and encode the pathological image blocks to obtain local pathological visual feature vectors. Virtual space registration and prediction module 202: used to acquire batch transcriptome sequencing data, and infer the virtual gene expression vector corresponding to each local pathological visual feature vector space through spatial mapping; and output the probability value of lymph node metastasis by performing risk prediction based on the local pathological visual feature vector and the virtual gene expression level. The spatial mapping involves using local pathological visual vectors as anchors to inversely map transcriptional sequencing data onto local pathological visual feature vectors to obtain corresponding virtual gene expression vectors.

[0020] The present invention also discloses a computer device, the device comprising: a memory and a processor; The memory is used to store program instructions; The processor is used to invoke program instructions, and when the program instructions are executed, the method steps are implemented.

[0021] The present invention also discloses a computer program product, including a computer program that is implemented by a processor to perform the method steps.

[0022] The present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method steps.

[0023] Advantages of this invention: The present invention has the following beneficial effects: 1. To address the problem that existing spatial transcriptome sequencing technologies rely on expensive physical sequencing chips, hindering large-scale retrospective studies and clinical application, this invention proposes a morphology-molecular manifold mapping algorithm. This algorithm uses local morphological features of pathological images as "anchors" to construct an energy minimization objective function that includes both global macro-constraints (based on cellular component deconvolution) and local micro-constraints (based on morphological prototype matching). By solving this objective function, this invention can inversely infer virtual gene expression vectors with spatial coordinate attributes using only conventional, low-cost batch transcriptome data and standard H&E pathological images without increasing hardware costs. This not only significantly reduces detection costs but also expands the analytical scope to conventional pathological archive samples, successfully constructing virtual gene expression maps with both high spatial resolution and molecular specificity.

[0024] 2. Conventional staining normalization methods based on generative adversarial networks often focus only on the transfer of global color style, easily neglecting the preservation of local texture structure. This invention constructs a fusion prediction model based on cross-gating units and tumor heterogeneity index correction. Virtual gene expression vectors are used as gating signals to filter meaningless morphological noise. The global pathological context vector is calculated through attention weights, and the weighted Euclidean distance variance in the feature space is further calculated to define a quantified heterogeneity index, which is directly used as a correction factor in the final calculation of lymph node metastasis probability. This model not only improves the specificity of feature extraction through a "gene-guided vision" mechanism but also innovatively transforms the abstract pathological concept of "tumor heterogeneity" into a calculable mathematical indicator. The higher the heterogeneity index, the greater the predicted metastasis risk, which is highly consistent with clinical pathological patterns, thus significantly improving the accuracy and biological interpretability of lymph node metastasis prediction. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic flowchart of an oral multimodal data fusion-assisted diagnosis method based on localized in-store computing provided by an embodiment of the present invention; Figure 2 This is a schematic flowchart illustrating the construction of an oral multimodal data fusion-assisted diagnostic model provided in an embodiment of the present invention. Figure 3 This is a pre-set OSCC single-cell gene expression reference map provided in the embodiments of the present invention; Figure 4This is a schematic diagram illustrating the determination of structural consistency loss provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of pseudocode for image preprocessing based on anatomical submersion guidance and structure preservation provided in an embodiment of the present invention; Figure 6 This is a pseudocode diagram of the virtual space registration algorithm based on morphological manifold constraints provided in an embodiment of the present invention; Figure 7 This is a schematic flowchart of an oral multimodal data fusion-assisted diagnostic system based on localized in-store computing provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the product interaction interface provided in an embodiment of the present invention; Figure 9 This is a schematic flowchart of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0028] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be performed in the order they appear herein, or may be performed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] This invention discloses a method for assisted diagnosis of oral multimodal data fusion based on localized in-store computing, see [link to relevant documentation]. Figure 1 ,include: S101: Acquire the full-field digital pathological image of the subject's oral cavity, divide it into blocks to obtain M pathological image blocks; encode the pathological image blocks to obtain local pathological visual feature vectors; Furthermore, encoding the pathological image block to obtain the local pathological visual feature vector specifically involves: inputting the pathological image block into a neural network for staining normalization processing, and extracting the normalized local pathological visual feature vector.

[0031] Optionally, the neural network includes any one or more of the following: generative adversarial networks, variational autoencoders; Optionally, the neural network is an improved generative adversarial network called StainGAN; the StainGAN network includes a mirror-symmetric generator and a discriminator.

[0032] Optionally, the specific configuration of the StainGAN network architecture is as follows: The generator employs a fully convolutional residual architecture and is configured to receive the original pathological image patch as input. It performs feature extraction and downsampling encoding on the input through multiple convolutional layers, mapping the image into a low-dimensional feature space. Stacked residual modules perform nonlinear coloring style transfer in the deep feature space; finally, the transferred features are upsampled and reconstructed through deconvolution layers to output the coloring-normalized target image patch.

[0033] Optionally, the discriminator employs a Markov discriminator architecture and is configured to receive the target image patch or standard template image patch as input, and map it to... The feature matrix, in which each element corresponds to the true / false discrimination result of a local receptive field patch in the input image patch, forces the generator to retain high-frequency texture details during the generation process.

[0034] S102: Acquire batch transcriptome sequencing data and infer the virtual gene expression vector corresponding to each of the local pathological visual feature vectors through spatial mapping; perform risk prediction and output the probability value of lymph node metastasis through local pathological visual feature vectors and virtual gene expression levels; wherein, the spatial mapping is to use the local pathological visual vector as the anchor point and reverse map the transcription sequencing data to the local pathological visual feature vector to obtain the corresponding virtual gene expression vector.

[0035] Furthermore, the transcriptome sequencing data is a gene expression matrix, and the local pathological visual feature vector includes cell morphology, tissue structure, and tissue texture features in the pathological image. The gene expression matrix is ​​inversely mapped to the cell morphology, tissue structure, and tissue texture features to obtain a virtual gene expression vector with spatial coordinate attributes.

[0036] Optionally, the reverse mapping process is as follows: the gene expression matrix is ​​deconvolutioned to calculate the cell abundance to obtain each cell type; the cell type is matched with the cell morphology, tissue structure, and tissue texture features in the local pathological visual feature vector.

[0037] Alternatively, the matching can be performed using a pre-built prototype library.

[0038] Optionally, the pre-built prototype library construction process is as follows: construct a first prototype vector set from the morphological features in the pathological image, average different morphological features in the first prototype vector set to obtain prototype vectors, each prototype vector representing a cell type, calculate the similarity between each image block of the pathological image and all prototype vectors, determine the cell type of each image block of the pathological image, and obtain the constructed prototype library.

[0039] Optionally, the spatial coordinate attribute is the spatial coordinates of image blocks mapped from cell type to pathological image.

[0040] Optionally, the spatial mapping is a morphology-molecular manifold mapping algorithm; the morphology-molecular manifold mapping algorithm specifically involves: constructing an energy minimization objective function, using the deconvolution result of the cellular components of the batch transcriptome sequencing data as a global macro-constraint, and using the morphological similarity matrix of the local pathological visual feature vectors as a local micro-constraint, and solving for a virtual gene expression vector with spatial coordinate attributes under the mutual constraints of the two.

[0041] In one specific embodiment, the specific process of this method is as follows (see...). Figure 2 First, load the batch transcriptome gene expression matrix from the sequencer. Subsequently, the system invokes a preset OSCC single-cell reference atlas (see...). Figure 3 This atlas is a standardized data matrix where rows represent genes, columns represent specific cell types within the OSCC microenvironment (e.g., tumor basal cells, fibroblasts, CD8+ T cells, macrophages, etc.), and values ​​represent the characteristic expression levels of specific genes in specific cell types. Based on this atlas, the system uses CIBERSORTx or support vector regression algorithms to analyze... Perform deconvolution calculation and output a cell abundance vector. ,in Representing the The relative proportion of each cell type in the entire tissue sample. In parallel, the output from S101 is received. A pathological image block Each image block The input is fed into a pre-trained ResNet-50 convolutional neural network, and its high-dimensional morphological feature vector is extracted. This vector It is a numerical representation of cell morphology, tissue structure, and texture within an image patch. The system internally maintains a prototype library containing... morphological prototype vectors Each prototype vector It was obtained by feature extraction and averaging of a large number of labeled single-cell type images, representing the first... The system identifies a visual "standard paradigm" or "average face" for each cell type (e.g., "tumor cells"). It calculates features for each image patch. All prototypes The cosine similarity between the images is used to determine which cell type the image patch is morphologically closest to.

[0042] In one specific embodiment, the system constructs and solves an energy minimization problem, taking global cell abundance as an example. As a macro-level constraint, the local morphological matching results are used as micro-level guidance for each image patch. Infer a virtual gene expression profile This solution process implements the mapping function, which will be detailed below. Input: Batch gene expression matrix ; A pathological image block Parallel processing: Molecular flow (right path): (Global cell scale vector); Morphology flow (left path): (Local visual feature vector); Core mapping: The system will... and This serves as input to the optimization solver. Based on the following energy function, the solver iteratively calculates the optimal set of virtual gene expression profiles. Output: A virtual gene expression profile with spatial coordinates It can be used to generate complex space heatmaps of any gene (such as COL1A1).

[0043] Furthermore, in the virtual space registration and inference process, the constructed spatial mapping inference energy function... (used for iteratively solving for virtual gene expression vectors of specific samples) It contains mutually restrictive local micro-constraints and global macro-constraints, and its specific mathematical expression is as follows: , in, Indicates the problem to be solved, the first... Virtual gene expression vectors for each pathological image patch This represents a batch transcriptome data matrix. This represents the regularization hyperparameter used to balance local micro-constraints and global macro-constraints. Indicates that based on morphological characteristics, it is the first The optimal gene expression prototype vector is dynamically matched to each image patch. The determination process includes: pre-constructing a system containing... A morphological prototype library of cell types was used to extract the first... High-dimensional morphological feature vector of a pathological image patch Calculate the high-dimensional morphological feature vector The average gene expression profile corresponding to the cell type with the highest cosine similarity to the morphological prototype vectors in the prototype library is selected as the mean similarity. , The representation of local morphological consistency constraints. It is an index function that returns an image patch. Morphological characteristics The best-matching cell prototype index. This represents the global constraint on the conservation of total biomass. This represents the regularization hyperparameter, a strong constraint that requires all inferred virtual representation spectra to be regularized. The sum must be exactly equal to the original batch transcriptome data. This ensures that the algorithm does not fabricate or lose biological information out of thin air, guaranteeing the biological fidelity of the results. A large This means that global conservation is more important than local shape matching, and vice versa. Theoretically... In embodiments of the present invention, the optimal value is determined by performing grid search and cross-validation on the validation set. The effective range is typically between [0.1, 10.0], and it was found that when At that time, the model achieved the best balance between the accuracy of the virtual spatial map and biological consistency.

[0044] In one specific embodiment, the optimal gene expression prototype vector The calculation process is as follows: based on morphological judgment, it is an image patch. The most suitable gene expression prototype was selected. This requirement mandates a virtual expression profile for each image patch. The gene expression profile should be as close as possible to the "standard" gene expression profile of the cell type that is most morphologically similar to it. For example, if an image patch looks like a "fibroblast," then its gene expression profile should be rich in genes related to collagen, etc. It is not a fixed string, but a dynamically selected prototype vector of gene expression based on morphological matching.

[0045] In one specific embodiment, the present invention employs a two-stream neural network architecture, comprising two parallel processing paths (see algorithm pseudocode reference). Figure 6 Visual feature flow: Input is the output of S101 A high-risk pathological image block sequence Local feature vectors for each image patch are extracted using a pre-trained ResNet-50 backbone network. , This refers to the feature dimension. Gene feature flow: The input is a virtual target expression vector generated by S102 for transfer-related genes such as the MMP and COL families. This vector is mapped to a global context vector through a multilayer perceptron. .

[0046] Furthermore, the method also includes prediction correction, which uses a virtual gene expression vector as a gating signal to weight and filter the local pathological visual feature vectors to generate a tumor heterogeneity index. The tumor heterogeneity index is used as a correction factor to adjust the risk prediction results, and the probability value of output lymph node metastasis is obtained. Optionally, a cross-gating unit is constructed, using the virtual gene expression vector as the gating signal to perform element-wise weighted filtering of the local pathological visual feature vectors of each pathological image block, in order to filter out morphological noise of no clinical significance; the calculation of the first... Attention weights of individual pathological image patches for predicting lymph node metastasis risk Based on the calculated attention weights For all input local pathological visual feature vectors By performing weighted summation, we obtain the pathological global context vector representing the overall characteristic distribution of the tumor. Calculate the weighted Euclidean distance variance of each local pathological visual feature vector relative to the global pathological context vector, and define it as the tumor heterogeneity index. .

[0047] Preferably, the attention weight The specific calculation formula is as follows: , in, and Indicates the input number of the first... The and the first A local pathological visual feature vector Represents the virtual gene expression vector. and Let represent the learnable projection matrices for visual features and genetic features, respectively. This represents element-wise multiplication. This represents the hyperbolic tangent nonlinear activation function. This represents the transpose of the attention scoring vector. This indicates the total number of high-risk pathological image blocks.

[0048] Optionally, the tumor heterogeneity index The specific calculation formula is as follows: , , in, Represents the global context vector of pathology. The squared L2 norm of a vector. The total number of image patches. Indicators of tumor heterogeneity, when The higher the value, the greater the difference in pathological features within the tumor.

[0049] This invention discloses an oral multimodal data fusion-assisted diagnostic system based on localized in-memory computing, see [link to relevant documentation]. Figure 7 The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the above-described method; The system is logically divided into: Preprocessing module 201: used to acquire the full-field digital pathological image of the subject's oral cavity, divide it into blocks to obtain M pathological image blocks; and encode the pathological image blocks to obtain local pathological visual feature vectors. Virtual space registration and prediction module 202: used to acquire batch transcriptome sequencing data, and infer the virtual gene expression vector corresponding to each local pathological visual feature vector space through spatial mapping; and output the probability value of lymph node metastasis by performing risk prediction based on the local pathological visual feature vector and the virtual gene expression level. The spatial mapping involves using local pathological visual vectors as anchors to inversely map transcriptional sequencing data onto local pathological visual feature vectors to obtain corresponding virtual gene expression vectors.

[0050] This embodiment also includes the client product form and interactive interface (see...). Figure 8The client software is divided into a product side (front-end interface) and a technical side (back-end logic). Product side interface design: The software interface mainly consists of three functional areas. The left side is the "Full-view Pathology Browsing Area," displaying high-resolution H&E-stained digital slides, supporting mouse wheel zooming and drag-and-drop navigation. Above the slide layers, the system uses a semi-transparent red heatmap to display high-risk areas identified by AI (such as the tumor invasion front). The right side is the "Multi-omics Association Analysis Area," displaying the expression levels of key genes (such as LIMCH1) in bar charts and volcano plots. When the user clicks on a region on the left-hand pathology image, the right side automatically refreshes to display the corresponding virtual anatomical gene expression profile. The bottom of the interface is the "Control and Reporting Area," displaying the "Lymph Node Metastasis Risk Score" (0-100 points) calculated by the model in real time, and providing a "One-Click Report Generation" button. Technical side back-end logic: The front-end is developed based on the Electron framework and communicates with the back-end Python service through a RESTful API. When a user imports data, the front end sends a POST request to trigger the back end's data cleaning task; when a user browses an image, the front end sends a GET request with coordinate parameters (x, y, zoom_level), and the back end's DeepZoom service dynamically cuts and returns the corresponding image tiles; when a user clicks the analysis button, the back end calls a PyTorch model for inference and returns the calculation results to the front end in JSON format for rendering.

[0051] The system provided in this embodiment is fully deployed on a local physical terminal (an integrated hardware and software machine), cutting off data transmission with external cloud servers to comply with medical data security and human genetic resource management regulations.

[0052] The specific hardware configuration is as follows: Processor (CPU): Employs a high-performance processor (such as Intel Core i7-14650HX or higher) for handling logical operations, data cleaning, and bioinformatics deconvolution calculations.

[0053] Graphics accelerator card (GPU): Equipped with a professional-grade graphics card (such as NVIDIA RTX A4000, with more than 16GB of video memory) to accelerate the training and inference of deep learning models (StainGAN and fusion prediction models).

[0054] Storage media: 4TB enterprise-grade NVMe SSD with built-in AES-256 hardware encryption module, used to store high-resolution full-view digital slices (WSI) and sequencing data.

[0055] Memory: 64GB DDR5, ensuring throughput for massive image tiles and large matrix operations.

[0056] The software architecture is based on a customized Linux operating system and integrates the Kubernetes container management platform. The backend uses a hybrid programming approach of Python (PyTorch framework) and C++ (OpenCV library), while the frontend uses the Electron framework to build a cross-platform user interface.

[0057] In one specific embodiment, the method flow follows a closed-loop logic of "input-cleaning-feature extraction-fusion-output", and the specific steps are as follows: Data acquisition and cleaning: The subjects' oral tissue full-view digital pathological images (WSI) and batch transcriptome sequencing data (Bulk RNA-seq, Fastq format) were received through an encrypted interface. Structure-preserving image preprocessing: To address the issues of varying staining depths across different medical institutions and the loss of micro-textures due to conventional normalization, this embodiment employs an improved StainGAN network to process WSI. Furthermore, for WSI images: the system calls the OpenCV library to convert the RGB image into a light density space, uses the Otsu algorithm to generate a binary mask, and automatically identifies and removes blank areas of the slide background; Furthermore, regarding the gene data: the FastQC tool was used to remove adapter sequences and low-quality bases from the sequencing data, and the STAR alignment software was used to align it to the human reference genome (hg38) to generate a gene expression matrix. .

[0058] In one specific embodiment, the image preprocessing process includes: block segmentation strategy, network structure, two-stream gradient constraint, and structural consistency guarantee.

[0059] The block segmentation strategy is specifically as follows (pseudocode reference) Figure 5 The process employs a multi-scale sliding window technique, segmenting the WSI into 512×512 pixel blocks with a stride of 512. The overlap rate between each image block and the mask in step S1 is calculated, retaining only valid image blocks with an overlap rate greater than 0.75. The network structure specifically includes: a generator employing a fully convolutional residual architecture, comprising 3 layers of downsampling encoding, 9 residual transformation modules, and 3 layers of upsampling decoding; and a discriminator employing a PatchGAN architecture, outputting a 70×70 feature matrix. The dual-stream gradient constraint ensures that during model inference, the generator not only outputs color-normalized images... It also computes the original image in parallel using the built-in Sobel operator layer. With generated image In horizontal and vertical Gradient plot of the direction.

[0060] This invention also discloses the system architecture and component interaction process. The system architecture includes a data layer, a service layer, and an interface layer. Data layer: Deployed on a local NVMe SSD, it includes a MongoDB database (storing user information and project metadata), MinIO object storage (storing unstructured WSI files and Fastq files), and a Redis cache (used to accelerate image tile retrieval). Service layer: Running in a Kubernetes cluster, it contains multiple microservice containers, including an "Image Processing Service" (running OpenCV and StainGAN), a "Bioinformatics Analysis Service" (running STAR and CIBERSORTx), an "AI Inference Service" (running ResNet and Attention Network), and a "Task Scheduling Service" (responsible for GPU resource allocation and load balancing). Interface layer: Provides HTTP / HTTPS protocol interfaces, handling requests from clients and performing authentication (JWT Token) and encrypted data transmission (SSL / TLS). The interaction process between the components is as follows: the client uploads data -> the interface layer receives and stores it in MinIO -> the task scheduling service detects new data and creates an analysis task -> the image processing service and bioinformatics analysis service read data from MinIO in parallel for processing -> intermediate results are stored in Redis -> the AI ​​inference service reads features from Redis for fusion calculation -> the final result is written to MongoDB -> the client polls to obtain the results and displays them.

[0061] This invention discloses a method for constructing an oral multimodal data fusion-assisted diagnostic model based on localized in-memory computing, comprising: S301: Acquire the subject's full-view digital pathological image of the oral cavity, batch transcriptome sequencing data, and disease status label, and input the digital pathological image and batch transcriptome sequencing data into the oral multimodal data fusion auxiliary diagnostic method based on local storage computing for processing to obtain virtual gene expression vectors; S302: Input the virtual gene expression vector into the prediction layer to obtain the prediction result; compare the prediction result with the disease status label to obtain the total loss function, and optimize the total loss function to obtain an oral multimodal data fusion-assisted diagnostic model based on localized storage computing.

[0062] In one specific embodiment, to address the problem that conventional GAN ​​models are prone to causing blurred cell nucleus edges or loss of texture during style transfer, an improved generative adversarial network (StainGAN) was deployed and implemented. This network integrates a two-stream gradient constraint mechanism on the basis of the classic CycleGAN cyclic consistency architecture.

[0063] (1) Specific architecture of the network model: The network core of this module consists of two mirror-symmetric generative adversarial paths (source domain to target domain, target domain to source domain), and the specific structural parameters are configured as follows: The generator employs a fully convolutional architecture based on ResNet-9. It includes three convolutional layers for feature downsampling encoding, nine residual modules for non-linear coloring style transfer in the deep feature space, and three deconvolutional layers for upsampling to restore image size. This structure ensures the complete transmission of deep semantic information.

[0064] The discriminator employs the PatchGAN architecture. Unlike traditional discriminators that output a single true / false probability value, this system's discriminator maps the input image to... The feature matrix of the image A local patch covering a 0-pixel area is used to distinguish between real and fake images. This design preserves high-frequency texture details and avoids image blurring during actual operation.

[0065] (2) Data flow and processing logic within the network: During the model training and inference phases, the data processing flow within the network strictly follows these steps (see...). Figure 2 ): Step 1: The forward generation flow system generates raw pathological image blocks. (Source Domain) Input Generator After encoding-conversion-decoding operations, a color-normalized image is output. (Target domain).

[0066] Step Two: Parallel Gradient Feature Extraction Flow. This is the key processing path of this invention. While generating the image, the system utilizes a built-in, non-trainable Sobel operator layer to process the original image in parallel. and generating images This layer is located along the horizontal ( ) and vertical ( The image is convolved in the direction of the image to extract the gradient magnitude maps of both images. This process extracts the edge contour information of the cell nucleus.

[0067] Step 3: The closed-loop feedback and parameter update system calculates the L1 distance between the gradient maps of the original and generated images to obtain the "structural consistency loss". This loss value is then weighted and summed with the "adversarial loss" and "cyclic consistency loss", and the generator's weight parameters are updated through the backpropagation algorithm.

[0068] In one specific embodiment, the total loss function is composed of a weighted sum of a structural consistency loss function, an adversarial loss function, and a cyclic consistency loss function.

[0069] Furthermore, the specific formula for the structural consistency loss function is as follows: ,in, This represents the mathematical expectation of the distribution of the original image. This represents the Sobel gradient operator operation, and the formula forces the generation of an image. The gradient distribution (i.e., the location and intensity of the cell nucleus edge) must be consistent with the original image. Maintaining pixel-level consistency. In actual testing, this constraint effectively prevents the generation of "artifacts" and ensures that the pathological diagnostic features do not undergo geometric distortion. The specific formula for the adversarial loss function is: , in, This represents the probability value of the discriminator classifying the real template image. This represents the probability value of the discriminator classifying the normalized image generated by the generator. This represents the mathematical expectation of the distribution of the standard template image. The adversarial loss function is used to make the generator output image approximate the standard template image in terms of color distribution. This enables the transfer of coloring styles.

[0070] Furthermore, the specific formula for the cycle consistency loss is (see...). Figure 4 ): , in, This represents the reconstructed image obtained after processing the normalized image by the inverse generator. The cycle consistency loss ensures that after the image is converted to the target style, it can still be restored back to the original image by the inverse generator, ensuring that the biological information of the image content is not lost.

[0071] Finally, the specific formula for the total loss function is as follows: ,in, and This is the adjustment coefficient for the loss function, and its value is a number greater than zero. Preferably, , .

[0072] In addition, the present invention also has the following beneficial effects: First, it accurately quantifies tumor heterogeneity to aid precision medicine. By introducing an attention mechanism and a specific weighting formula, this invention can automatically quantify the contribution of different tumor regions (such as the central region and the invasion front) to the risk of metastasis. Compared to the qualitative description of traditional manual image reading, this system provides a quantitative risk score, effectively solving the problem of poor diagnostic consistency caused by tumor heterogeneity.

[0073] Second, it improves the detection rate of tiny, occult lesions. Utilizing multi-scale pyramid slicing technology, the system can scan every high-power field of view without omission. Combined with the high sensitivity of deep learning models to microscopic features such as "tumor budding," it effectively avoids the missed diagnosis of tiny metastases caused by fatigue or limited field of view during manual image reading, increasing the prediction accuracy to over 90%.

[0074] Third, it achieves logical mutual verification of cross-modal information. By combining morphological features with molecular markers through a fusion formula, a dual verification mechanism of "image-gene" is systematically constructed. When pathological image features are atypical, highly expressed invasion-related genes (such as COL1A1) can serve as a supplementary risk indication; and vice versa. This mutual verification mechanism significantly improves the specificity of diagnosis.

[0075] Fourth, it balances data security and analytical efficiency. Based on the localized all-in-one machine architecture, all complex convolution operations and matrix calculations are performed on the local GPU, eliminating the need for data to leave the hospital and strictly complying with regulations such as the "Regulations on the Management of Human Genetic Resources." Simultaneously, the high-performance hardware compresses the analysis process, which originally took several days, to within 8 hours, meeting the clinical need for rapid intraoperative or postoperative decision-making.

[0076] This invention also discloses a device for constructing a multi-omics disease prediction model, the device comprising: a memory and a processor, such as... Figure 9 As shown: The memory is used to store program instructions; The processor is used to invoke program instructions, and when the program instructions are executed, the method steps are implemented.

[0077] The present invention provides a computer program product, including a computer program, which is implemented by a processor to implement the method steps.

[0078] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method steps.

[0079] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0081] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0082] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The aforementioned integrated modules can be implemented in hardware or as software functional modules.

[0083] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0084] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0085] The computer device provided by the present invention has been described in detail above. For those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for assisted diagnosis of oral multimodal data fusion based on localized in-store computing, characterized in that, This method is entirely deployed and executed on a local physical terminal, including: S101: Acquire the full-field digital pathological image of the subject's oral cavity, divide it into blocks to obtain M pathological image blocks; encode the pathological image blocks to obtain local pathological visual feature vectors; S102: Acquire batch transcriptome sequencing data, and infer the virtual gene expression vector corresponding to each local pathological visual feature vector space through spatial mapping; perform risk prediction and output the probability value of lymph node metastasis through local pathological visual feature vectors and virtual gene expression levels; The spatial mapping involves using local pathological visual vectors as anchors to inversely map transcriptional sequencing data onto local pathological visual feature vectors to obtain corresponding virtual gene expression vectors.

2. The oral multimodal data fusion-assisted diagnosis method based on localized in-store computing according to claim 1, characterized in that, The transcriptome sequencing data is a gene expression matrix, and the local pathological visual feature vector includes cell morphology, tissue structure, and tissue texture features in the pathological image. The gene expression matrix is ​​inversely mapped to the cell morphology, tissue structure, and tissue texture features to obtain a virtual gene expression vector with spatial coordinate attributes. Optionally, the reverse mapping process is as follows: the gene expression matrix is ​​deconvolutioned to calculate the cell abundance to obtain each cell type; the cell type is matched with the cell morphology, tissue structure, and tissue texture features in the local pathological visual feature vector; Optionally, the matching is performed using a pre-built prototype library; Optionally, the pre-built prototype library construction process is as follows: construct a first prototype vector set from the morphological features in the pathological image, average the different morphological features in the first prototype vector set to obtain prototype vectors, each prototype vector represents a cell type, calculate the similarity between each image block of the pathological image and all prototype vectors, determine the cell type of each image block of the pathological image, and obtain the constructed prototype library. Optionally, the spatial coordinate attribute is the spatial coordinates of image patches mapped from cell type to pathological image; The spatial mapping is a morphology-molecular manifold mapping algorithm; the morphology-molecular manifold mapping algorithm specifically involves: constructing an energy minimization objective function, using the deconvolution result of the cellular components of the batch transcriptome sequencing data as a global macro-constraint, and using the morphological similarity matrix of the local pathological visual feature vectors as a local micro-constraint, and solving for the virtual gene expression vector with spatial coordinate attributes under the mutual constraints of the two.

3. The oral multimodal data fusion-assisted diagnosis method based on localized in-store computing according to claim 2, characterized in that, The specific steps of encoding the pathological image block to obtain the local pathological visual feature vector are as follows: inputting the pathological image block into a neural network for staining normalization processing, and extracting the normalized local pathological visual feature vector. Optionally, the neural network includes any one or more of the following: generative adversarial networks, variational autoencoders; Optionally, the neural network is an improved generative adversarial network called StainGAN; the StainGAN network includes a mirror-symmetric generator and a discriminator. Optionally, the specific configuration of the StainGAN network architecture is as follows: The generator employs a fully convolutional residual architecture and is configured to receive the original pathological image patch as input. It performs feature extraction and downsampling encoding on the input through multiple convolutional layers, mapping the image into a low-dimensional feature space. Stacked residual modules perform nonlinear coloring style transfer in the deep feature space; finally, the transferred features are upsampled and reconstructed through deconvolution layers to output the coloring-normalized target image patch. The discriminator employs a Markov discriminator architecture and is configured to receive the target image patch or standard template image patch as input, and map it to... The feature matrix, in which each element corresponds to the true / false discrimination result of a local receptive field patch in the input image patch, forces the generator to retain high-frequency texture details during the generation process.

4. The oral multimodal data fusion-assisted diagnosis method based on localized in-store computing according to claim 2, characterized in that, The constructed energy minimization objective function includes mutually constraining local micro-constraints and global macro-constraints. The specific mathematical expression of the energy minimization objective function is as follows: , in, Indicates the problem to be solved, the first... Virtual gene expression vectors for each pathological image patch This represents a batch transcriptome data matrix. This represents the regularization hyperparameter used to balance local micro-constraints and global macro-constraints. Indicates that based on morphological characteristics, it is the first The optimal gene expression prototype vector is dynamically matched to each image patch. The determination process includes: pre-constructing a system containing... A morphological prototype library of cell types was used to extract the first... High-dimensional morphological feature vector of a pathological image patch Calculate the high-dimensional morphological feature vector The average gene expression profile corresponding to the cell type with the highest cosine similarity to the morphological prototype vectors in the prototype library is selected as the mean similarity. .

5. The oral multimodal data fusion-assisted diagnosis method based on localized in-store computing according to claim 1, characterized in that, include: The method further includes prediction correction, which uses a virtual gene expression vector as a gating signal to weight and filter the local pathological visual feature vectors to generate a tumor heterogeneity index. The tumor heterogeneity index is used as a correction factor to adjust the risk prediction results, and the probability value of output lymph node metastasis is obtained. Optionally, a cross-gating unit is constructed, using the virtual gene expression vector as the gating signal to perform element-wise weighted filtering of the local pathological visual feature vectors of each pathological image block, in order to filter out morphological noise of no clinical significance; the calculation of the first... Attention weights of individual pathological image patches for predicting lymph node metastasis risk Based on the calculated attention weights For all input local pathological visual feature vectors By performing weighted summation, we obtain the pathological global context vector representing the overall characteristic distribution of the tumor. Calculate the weighted Euclidean distance variance of each local pathological visual feature vector relative to the global pathological context vector, and define it as the tumor heterogeneity index. .

6. The oral multimodal data fusion-assisted diagnosis method based on localized in-store computing according to claim 5, characterized in that, The attention weight The specific calculation formula is as follows: ， in, and Indicates the input number of the first... The and the first A local pathological visual feature vector Represents the virtual gene expression vector. and Let represent the learnable projection matrices for visual features and genetic features, respectively. This represents element-wise multiplication. This represents the hyperbolic tangent nonlinear activation function. This represents the transpose of the attention scoring vector. This indicates the total number of high-risk pathological image blocks; Optionally, the tumor heterogeneity index The specific calculation formula is as follows: ， ， in, Represents the global context vector of pathology. The squared L2 norm of a vector. The total number of image patches. Indicators of tumor heterogeneity, when The higher the value, the greater the difference in pathological features within the tumor.

7. A localized in-store computing-based multimodal oral data fusion-assisted diagnostic system, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-6; The system is logically divided into: Preprocessing module 201: used to acquire the full-field digital pathological image of the subject's oral cavity, divide it into blocks to obtain M pathological image blocks; and encode the pathological image blocks to obtain local pathological visual feature vectors. Virtual space registration and prediction module 202: used to acquire batch transcriptome sequencing data, and infer the virtual gene expression vector corresponding to each local pathological visual feature vector space through spatial mapping; and output the probability value of lymph node metastasis by performing risk prediction based on the local pathological visual feature vector and the virtual gene expression level. The spatial mapping involves using local pathological visual vectors as anchors to inversely map transcriptional sequencing data onto local pathological visual feature vectors to obtain corresponding virtual gene expression vectors.

8. A computer device, comprising: Memory and processor; The memory is used to store program instructions; The processor is used to invoke program instructions, which, when executed, implement the method according to any one of claims 1-7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1-7.

Images