Sperm cell analysis and diagnosis system based on multi-modal large language model

Abstract

The invention relates to a sperm cell analysis and diagnosis system based on a multi-modal large language model, which comprises a multi-modal data co-processing unit, a cross-modal semantic alignment module, a dynamic diagnosis decision engine and a self-adaptive evolution system, the four-dimensional data processing module is used for synchronously processing microscopic images, motion trail videos, biochemical detection data and four-dimensional input data of medical record texts and comprises a feature selector based on a gating attention mechanism. According to the sperm cell analysis and diagnosis system based on the multi-modal large language model, quantitative analysis of sperm movement chaos features is realized for the first time, a nonlinear dynamic evaluation standard is established, a cross-modal knowledge distillation and meta-learning migration framework is developed, a data annotation dependence bottleneck is broken through, and an interpretable clinical decision support system is constructed; dynamic updating and probabilistic suggestion of diagnosis rules are achieved, semantic analysis of single-cell multi-omics data is achieved, and molecular mechanism research results are converted into clinically available knowledge.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of biomedical engineering and artificial intelligence, specifically to a sperm cell analysis and diagnostic system based on a multimodal large language model. Background Technology

[0002] Current sperm analysis technology in the field of male reproductive health faces significant technical bottlenecks in multimodal fusion, dynamic modeling, and intelligent diagnosis, specifically manifested in the following five dimensions of technical deficiencies:

[0003] I. Technical Challenges of Multimodal Data Integration

[0004] Data modal isolation

[0005] Existing systems (such as patent CN118506363A) only achieve simple splicing of microscopic images and basic motion parameters, without establishing semantic association between molecular markers (such as DNA fragmentation index) and morphological features (such as acrosome integrity).

[0006] The lack of spatiotemporal alignment algorithms for flow cytometry data (such as CD54 expression levels) and temporal characteristics of movement trajectories makes it impossible to resolve the causal relationship of "surface antigen expression → change in movement pattern".

[0007] Cross-modal knowledge transfer barriers

[0008] Traditional deep learning models (such as patent CN118196406A) employ an independent training strategy, which fails to map the biological significance of "PRM1 gene mutation" in gene sequencing data to the image feature space.

[0009] Existing multimodal fusion methods (such as early fusion / late fusion) ignore the heterogeneous characteristics of different modal data. For example, the temporal nature of motion trajectories and the spatial nature of image data are not jointly modeled through a unified spatiotemporal Transformer network.

[0010] II. Technical Limitations of Sperm Motility Analysis

[0011] Insufficient dynamic process modeling

[0012] Existing trajectory analysis algorithms (such as patent CN117455958A) are based on second-order equations of motion and cannot capture the chaotic characteristics of sperm movement (such as nonlinear motion with a Lyapunov exponent > 0.5).

[0013] The lack of phase analysis of the motion trajectory makes it impossible to identify periodic motion anomalies (such as periodic jitter of the flagellar waving frequency).

[0014] Motion-morphology association missing

[0015] Traditional methods assume that motion parameters (such as VSL) are linearly related to morphological features (such as tail length), but in reality there is nonlinear coupling (such as the spatial correlation between tail bending angle and mitochondrial distribution).

[0016] III. Precision Bottlenecks in Morphological Analysis

[0017] Insufficient ability to analyze ultrastructures

[0018] Existing microscopic imaging techniques (such as bright-field microscopy) have a resolution limited to 0.2 μm, making it impossible to identify nanoscale defects in the acrosome film (such as cavitation with a diameter of <100 nm).

[0019] Fluorescent staining methods (such as Hoechst staining) only provide two-dimensional information and lack the ability to reconstruct three-dimensional structures, making it impossible to assess the spatial heterogeneity of nuclear chromatin condensation.

[0020] Defects in the abnormal pattern detection mechanism

[0021] Supervised learning-based morphological classification models (such as patent CN114219071A) rely on known abnormal templates and cannot automatically identify novel malformation types (such as specific tail curling).

[0022] IV. Technical Shortcomings of Intelligent Assisted Diagnosis

[0023] Clinical knowledge embedding deficiency

[0024] The lack of a dynamic probabilistic reasoning framework makes it impossible to quantify the contribution of different features to diagnostic results (such as the combined effect coefficient of morphological normality rate and DNA fragmentation rate).

[0025] Weak Explanation Mechanisms

[0026] The black-box nature of deep learning models makes it difficult for clinicians to understand the decision-making logic. For example, they may be unable to locate the connection between key image regions (such as the acrosome region) that the model focuses on and specific molecular mechanisms.

[0027] V. Technical Challenges of Zero-Shot Prediction

[0028] Insufficient generalization ability of new data

[0029] Existing models (such as patent CN113920070A) rely on a large amount of labeled data and need to be retrained when faced with new detection technologies (such as single-cell proteomics data), making it impossible to achieve zero-sample adaptation.

[0030] Lacking a meta-learning mechanism, it is difficult to quickly adapt to differences in different laboratory equipment (such as differences in image style caused by changes in objective magnification).

[0031] Cross-species knowledge transfer barriers

[0032] The characteristic space of human sperm differs significantly from that of model organisms (such as mice), and existing models cannot utilize existing animal model data (such as gene knockout mouse sperm databases) to improve the accuracy of human diagnosis.

[0033] Therefore, there is an urgent need to propose a sperm cell analysis and diagnostic system based on a multimodal large language model to solve the above problems. Summary of the Invention

[0034] To address the shortcomings of existing technologies, this invention provides a sperm cell analysis and diagnostic system based on a multimodal large language model. This technology integrates computer vision, natural language processing, biomedical engineering, and deep learning technologies to construct a multi-dimensional data fusion analysis platform. This platform enables automated analysis of multimodal data such as sperm cell morphological characteristics, motility trajectories, and molecular markers, providing an innovative solution for the accurate diagnosis and personalized treatment of male infertility and reproductive system diseases. It solves the problems of multimodal data fusion difficulties, dynamic process modeling defects, morphological analysis accuracy bottlenecks, and insufficient intelligent diagnostic capabilities in traditional sperm cell analysis and diagnostic systems.

[0035] To achieve the above objectives, the present invention provides the following technical solution: a sperm cell analysis and diagnosis system based on a multimodal large language model, comprising a multimodal data collaborative processing unit, a cross-modal semantic alignment module, a dynamic diagnostic decision engine, and an adaptive evolutionary system:

[0036] The multimodal data collaborative processing unit is used to simultaneously process four-dimensional input data including microscopic images, motion trajectory videos, biochemical test data, and medical record texts, and includes:

[0037] A feature selector based on a gated attention mechanism is used to achieve dynamic weighted extraction of key features;

[0038] A 3D CNN-Transformer hybrid architecture for modeling and analyzing temporal motion features;

[0039] The cross-modal semantic alignment module maps heterogeneous data to a unified semantic space, and includes:

[0040] A dual-tower contrastive learning framework achieves image-text alignment through visual and text encoding towers.

[0041] A multi-scale feature pyramid was used to establish a correlation matrix between cell morphology features and clinical indicators.

[0042] The dynamic diagnostic decision engine constructs a hierarchical reasoning mechanism with clinical interpretability, including a three-stage decision-making process: morphological abnormality detection → motor function assessment → comprehensive fertility index calculation. It develops a dynamic threshold adjustment module based on reinforcement learning, automatically optimizes diagnostic criteria according to contextual information such as patient age and medical history, realizes the visual traceability of diagnostic basis, locates abnormal cell regions through heat maps, and generates natural language reports that conform to clinical norms.

[0043] The adaptive evolution system establishes a model update mechanism under a federated learning framework, supports the privacy and security sharing of multi-center medical data, designs a drift detection algorithm to monitor model performance degradation, triggers online incremental learning to maintain diagnostic accuracy, constructs a knowledge graph-driven abnormal case learning module, and mines potential feature patterns of rare diseases through graph neural networks.

[0044] Preferably, the attention weights of the feature selector are calculated to satisfy:

[0045] α_i=σ(W_q·[f_i||c]+b)

[0046] Where f_i is the i-th feature vector, c is the clinical context vector, and W_q and b are learnable parameters;

[0047] The 3D CNN-Transformer hybrid architecture includes:

[0048] The SigLIP visual feature extraction model is used to extract contextual features from high-resolution images.

[0049] InternVL 3.0 is a multimodal large language model that extracts multimodal and language capabilities from diverse multimodal data and plain text corpora.

[0050] Preferably, the cross-modal semantic alignment module includes:

[0051] The visual coding tower adopts a multi-task learning framework to simultaneously output sperm head morphology classification results and tail defect detection heatmaps.

[0052] Text encoding tower extracts key values ​​of patients' reproductive history and seminal plasma biochemical indicators from medical records through medical entity recognition (NER);

[0053] The loss function for contrastive learning is the improved NT-Xent loss:

[0054] L_{cont}=-log[exp(sim(v,t) / τ) / (∑_{k≠t}exp(sim(v,k) / τ)]

[0055] Where v represents visual features, t represents matching text features, and k represents negative samples.

[0056] Preferably, the dynamic diagnostic decision engine includes:

[0057] The three-stage reasoning submodule sequentially performs anomaly detection based on morphological features, vitality assessment combined with motion trajectory parameters, and calculation of a comprehensive fertility index integrating clinical history.

[0058] The threshold dynamic adjustment submodule employs a reinforcement learning agent, whose state space is defined as follows:

[0059] s_t = [age, abnormality rate, forward motor velocity, severity of medical history]

[0060] The action space is ±10% of the diagnostic threshold.

[0061] Furthermore, in the three-stage reasoning submodule:

[0062] The output of morphological anomaly detection includes the head cavitation ratio, acrosome integrity score, and tail bending angle statistics.

[0063] Vitality assessment uses a weighted formula based on exercise parameters:

[0064] Motility Score=0.6×VCL+0.3×ALH+0.1×BCF

[0065] Where VCL is the curve velocity, ALH is the head lateral swing amplitude, and BCF is the whipping frequency.

[0066] Preferably, the adaptive evolutionary system:

[0067] The federated learning framework employs a gradient aggregation protocol based on homomorphic encryption;

[0068] Knowledge graphs contain three types of entity relations:

[0069] (Types of sperm abnormalities) - [requires] -> (ICSI treatment);

[0070] (Oxidative stress) - [affects] -> (DNA fragmentation rate);

[0071] The drift detection module monitors changes in the distribution of input data using KL divergence.

[0072] Furthermore, it also includes:

[0073] The report generation module automatically outputs diagnostic reports that conform to WHO standards, including heatmaps of abnormal cell locations, animations reconstructing their movement trajectories, and treatment recommendations described in natural language.

[0074] Furthermore, it also includes the use of a systematic sperm cell analysis method, including the following steps:

[0075] 1) Extract depth features from microscopic images and motion videos using a multimodal data collaborative processing unit;

[0076] 2) Establish the association between image features and medical record text using a cross-modal semantic alignment module;

[0077] 3) The dynamic diagnostic decision engine generates hierarchical diagnostic conclusions;

[0078] 4) Update model parameters through an adaptive evolutionary system.

[0079] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0080] 1. This sperm cell analysis and diagnostic system based on a multimodal large language model constructs a spatiotemporal-semantic joint Transformer network to achieve cross-modal deep alignment of molecular markers, dynamic trajectories, and ultrastructures. It establishes a sperm knowledge graph containing 200+ entity relationships, enabling cross-scale reasoning from "morphological abnormalities → molecular pathway abnormalities → clinical phenotypes." The correlation coefficient of multimodal features is improved, a contrastive learning-driven abnormality detection model is developed, and combined with super-resolution microscopy, three-dimensional reconstruction of sperm ultrastructure is achieved, improving the automatic recognition rate. Three new types of malformations are discovered, and the pathological feature database not covered in the WHO standards is expanded.

[0081] 2. This sperm cell analysis and diagnostic system based on a multimodal large language model introduces meta-learning and domain adaptation technologies, constructs a cross-species knowledge transfer framework, improves the adaptation speed to single-cell proteomics data, achieves a zero-sample prediction accuracy of 87%, enhances cross-laboratory data generalization ability, reduces diagnostic bias caused by equipment differences, introduces Lyapunov index analysis and spatiotemporal Transformer prediction models, establishes sperm motility stability assessment standards, discovers dynamic abnormalities that traditional parameters cannot characterize, reduces motion trajectory prediction errors, achieves a 50ms trajectory prediction accuracy of 89%, constructs a differentiable clinical decision tree and Bayesian inference framework, develops attention heatmap visualization technology, improves diagnostic consistency, increases the transparency of decision-making logic, generates counterfactual explanations, and assists doctors in understanding the comparison table of diagnostic basis technical indicators.

[0082] 3. This sperm cell analysis and diagnostic system based on a multimodal large language model achieves, for the first time, the quantitative analysis of chaotic features of sperm motility, establishes nonlinear dynamics evaluation standards, develops a cross-modal knowledge distillation and meta-learning transfer framework, breaks through the bottleneck of data labeling dependence, constructs an interpretable clinical decision support system, realizes dynamic updating and probabilistic suggestions of diagnostic rules, realizes semantic parsing of single-cell multi-omics data, and transforms molecular mechanism research results into clinically usable knowledge. Attached Figure Description

[0083] Figure 1 This is a block diagram of the sperm cell analysis and diagnosis system based on a multimodal large language model according to the present invention;

[0084] Figure 2 This invention presents a three-stage reasoning process for a sperm cell analysis and diagnosis system based on a multimodal large language model.

[0085] Figure 3 This is a schematic diagram of Embodiment 2 of the sperm cell analysis and diagnosis system based on a multimodal large language model of the present invention. Detailed Implementation

[0086] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] Example 1

[0088] Please see Figure 1-2 The sperm cell analysis and diagnosis system based on a multimodal large language model in this embodiment includes a multimodal data collaborative processing unit, a cross-modal semantic alignment module, a dynamic diagnostic decision engine, and an adaptive evolutionary system.

[0089] A multimodal data collaborative processing unit is used to simultaneously process four-dimensional input data, including microscopic images, motion trajectory videos, biochemical test data, and medical record texts, and includes:

[0090] A feature selector based on a gated attention mechanism is used to achieve dynamic weighted extraction of key features;

[0091] A 3D CNN-Transformer hybrid architecture for modeling and analyzing temporal motion features.

[0092] The cross-modal semantic alignment module maps heterogeneous data to a unified semantic space, and includes:

[0093] A dual-tower contrastive learning framework achieves image-text alignment through visual and text encoding towers.

[0094] A multi-scale feature pyramid was used to establish a correlation matrix between cell morphology features and clinical indicators.

[0095] The attention weights of the feature selector are calculated to satisfy:

[0096] α_i=σ(W_q·[f_i||c]+b)

[0097] Where f_i is the i-th feature vector, c is the clinical context vector, and W_q and b are learnable parameters;

[0098] The 3D CNN-Transformer hybrid architecture includes:

[0099] The SigLIP visual feature extraction model is used to extract contextual features from high-resolution images.

[0100] InternVL 3.0 is a multimodal large language model that extracts multimodal and language capabilities from diverse multimodal data and plain text corpora.

[0101] The cross-modal semantic alignment module includes:

[0102] The visual coding tower adopts a multi-task learning framework to simultaneously output sperm head morphology classification results and tail defect detection heatmaps.

[0103] Text encoding tower extracts key values ​​of patients' reproductive history and seminal plasma biochemical indicators from medical records through medical entity recognition (NER);

[0104] The loss function for contrastive learning is the improved NT-Xent loss:

[0105] L_{cont}=-log[exp(sim(v,t) / τ) / (∑_{k≠t}exp(sim(v,k) / τ)]

[0106] Where v represents visual features, t represents matching text features, and k represents negative samples.

[0107] The dynamic diagnostic decision engine constructs a hierarchical reasoning mechanism with clinical interpretability, including a three-stage decision-making process: morphological abnormality detection → motor function assessment → comprehensive fertility index calculation. It develops a dynamic threshold adjustment module based on reinforcement learning, automatically optimizes diagnostic criteria according to contextual information such as patient age and medical history, realizes the visual traceability of diagnostic basis, locates abnormal cell regions through heat maps and generates natural language reports that conform to clinical norms.

[0108] The dynamic diagnostic decision engine includes:

[0109] The three-stage reasoning submodule sequentially performs anomaly detection based on morphological features, vitality assessment combined with motion trajectory parameters, and calculation of a comprehensive fertility index integrating clinical history.

[0110] The threshold dynamic adjustment submodule employs a reinforcement learning agent, whose state space is defined as follows:

[0111] s_t = [age, abnormality rate, forward motor velocity, severity of medical history]

[0112] The action space is ±10% of the diagnostic threshold.

[0113] Furthermore, in the three-stage reasoning submodule:

[0114] The output of morphological anomaly detection includes the head cavitation ratio, acrosome integrity score, and tail bending angle statistics.

[0115] Vitality assessment uses a weighted formula based on exercise parameters:

[0116] Motility Score=0.6×VCL+0.3×ALH+0.1×BCF

[0117] Where VCL is the curve velocity, ALH is the head lateral swing amplitude, and BCF is the whipping frequency.

[0118] An adaptive evolutionary system establishes a model update mechanism within a federated learning framework, supports the privacy-preserving and secure sharing of multi-center medical data, designs a drift detection algorithm to monitor model performance degradation, triggers online incremental learning to maintain diagnostic accuracy, constructs a knowledge graph-driven anomalous case learning module, and mines latent feature patterns of rare diseases through graph neural networks. The adaptive evolutionary system includes:

[0119] The federated learning framework employs a gradient aggregation protocol based on homomorphic encryption;

[0120] Knowledge graphs contain three types of entity relations:

[0121] (Types of sperm abnormalities) - [requires] -> (ICSI treatment);

[0122] (Oxidative stress) - [affects] -> (DNA fragmentation rate);

[0123] The drift detection module monitors changes in the distribution of input data using KL divergence.

[0124] In this embodiment, the report generation module automatically outputs a diagnostic report that conforms to WHO standards, including a heatmap of the location of abnormal cells, a reconstructed animation of their movement trajectories, and treatment recommendations described in natural language.

[0125] In this embodiment, the sperm cell analysis method of the system includes the following steps:

[0126] 1) Extract depth features from microscopic images and motion videos using a multimodal data collaborative processing unit;

[0127] 2) Establish the association between image features and medical record text using a cross-modal semantic alignment module;

[0128] 3) The dynamic diagnostic decision engine generates hierarchical diagnostic conclusions;

[0129] 4) Update model parameters through an adaptive evolutionary system.

[0130] Example 2

[0131] Please see Figure 3 In this embodiment, the construction of the sperm cell analysis and diagnostic system based on a multimodal large language model is carried out in the following steps:

[0132] I. System Overall Architecture

[0133] The sperm cell analysis and diagnosis system based on a multimodal large language model provided by this invention mainly includes a multimodal data acquisition module, a preprocessing module, a multimodal large language model analysis module, a diagnosis module, and a user interaction module. The modules communicate and exchange data via a data bus, working together to complete the analysis and diagnosis of sperm cells.

[0134] II. Detailed Structure and Interrelationships of Each Module

[0135] (I) Multimodal data acquisition module

[0136] 1. Hardware Components

[0137] This module consists of a microscopic imaging device, a sensor array, and a text input device. The microscopic imaging device includes a high-resolution microscope (adjustable magnification from 400-1000x), a CCD camera (resolution no less than 1920×1080), and a stage (with three-dimensional motorized adjustment capability and an adjustment accuracy of 1μm), used to acquire microscopic images and dynamic motion videos of sperm cells. The sensor array includes a temperature sensor (accuracy ±0.1℃), a pH sensor (accuracy ±0.01), and an osmotic pressure sensor (accuracy ±1mOsm / kg), used to collect environmental parameters of the semen sample. The text input device can be a keyboard, scanner, or voice input device, used to input basic patient information, medical history, semen collection time, and other textual data.

[0138] 2. Circuit Structure

[0139] Each hardware device connects to the main control circuit board via a data interface. The CCD camera transmits image and video data to the main control circuit board via a USB 3.0 interface; the sensor group converts analog signals into digital signals via an ADC conversion module before transmitting them to the main control circuit board; the text input device connects to the main control circuit board via a standard input interface (such as a USB interface). The main control circuit board uses an ARM architecture microprocessor, which has high-speed data processing and transmission capabilities. It temporarily stores the acquired multimodal data in a cache chip and transmits it to the preprocessing module via an Ethernet interface.

[0140] (II) Preprocessing Module

[0141] 1. Hardware Components

[0142] The preprocessing module is mainly composed of a high-performance computer with the following configuration: Intel Core i7 processor (3.6GHz or higher), 32GB or more of memory, 512GB or more of solid-state drive, and NVIDIA RTX 3080 or higher dedicated graphics card, used to run data preprocessing algorithms.

[0143] 2. Software Structure and Processing Flow

[0144] Image data received from the multimodal data acquisition module undergoes initial denoising, employing median filtering to remove salt-and-pepper noise, followed by contrast enhancement using histogram equalization. For video data, frame extraction decomposes the dynamic video into a series of single frames, each then undergoes the same denoising and enhancement processes as static images. Text data is cleaned, removing invalid characters and duplicate information, and then standardized to convert different formats into a unified format, such as extracting and storing medical history information according to preset fields. Sensor data is calibrated, correcting the acquired data based on sensor calibration parameters, and then normalized to convert sensor data of different dimensions into values ​​between 0 and 1. The preprocessed multimodal data is then transmitted to the multimodal large language model analysis module via a data interface.

[0145] (III) Multimodal Large Language Model Analysis Module

[0146] 1. Hardware Components

[0147] This module consists of a server cluster, which includes multiple high-performance servers. Each server is configured with: an Intel Xeon processor (2.8GHz or higher), 64GB or more of RAM, 1TB or more of SSD storage, and an NVIDIA A100 or higher accelerated graphics card. The servers are connected via high-speed network switches to form a distributed computing environment for running multimodal large language models.

[0148] 2. Model Architecture and Processing Procedure

[0149] The multimodal large language model employs a Transformer architecture, comprising an image encoder, a text encoder, a sensor data encoder, and a fusion decoder. The image encoder uses a convolutional neural network (CNN) to extract features from preprocessed sperm cell images, including sperm morphological features such as head size, tail length, and deformity rate. The text encoder uses a deep learning-based pre-trained RoBERTa encoder to extract semantic features from standardized text data, including patient age, medical history, and semen collection conditions. The sensor data encoder uses a fully connected neural network to extract features from normalized sensor data, including environmental parameters such as semen temperature, pH, and osmotic pressure.

[0150] The feature vectors from the three modalities are fused through a fusion layer. An early fusion strategy is employed, concatenating image features, text features, and sensor data features dimensionally to form a multimodal fused feature vector. This fused feature vector is then input into the decoder of the large language model, where it is analyzed using a self-attention mechanism and a feedforward neural network to learn the mapping relationship between sperm cell features and sperm quality indicators. During training, the model utilizes a large amount of labeled data for supervised learning. This labeled data includes microscopic images of sperm cells, corresponding text information, sensor data, and diagnostic results of sperm quality (such as sperm concentration, motility, and morphological normality). The training algorithm employs the Adam optimization algorithm, with a loss function that is a combination of mean squared error loss and cross-entropy loss.

[0151] (iv) Diagnostic Module

[0152] 1. Hardware Components

[0153] The diagnostic module shares the hardware resources of the server cluster with the multimodal large language model analysis module, and uses the server's computing power to reason about diagnostic rules and generate diagnostic results.

[0154] 2. Software Structure and Diagnostic Process

[0155] The diagnostic module comprises a diagnostic rule knowledge base and an inference engine. The diagnostic rule knowledge base stores standard rules and clinical experience in the medical field regarding sperm cell analysis and diagnosis, such as the sperm quality assessment standards established by the World Health Organization (WHO). The inference engine performs inference based on the sperm quality index prediction results output by the multimodal large language model analysis module, combined with the rules in the diagnostic rule knowledge base.

[0156] The specific process is as follows: First, the sperm concentration, motility, and morphological normality rate output by the large language model are compared with WHO standards to determine whether each indicator is normal. Then, combined with the patient's medical history (such as whether there are reproductive system diseases or exposure to harmful substances) and semen collection environmental parameters (such as whether the temperature and collection time meet requirements), a comprehensive analysis of possible causes of abnormal sperm quality is conducted. Finally, preliminary diagnostic suggestions are generated, such as whether further examination is needed and whether there are risks affecting fertility. The diagnostic results are transmitted to the user interaction module via a data interface.

[0157] (V) User Interaction Module

[0158] 1. Hardware Components

[0159] The user interaction module includes a monitor (resolution no less than 1920×1080), a keyboard, a mouse, and a printer. The monitor is used to display images, videos, analysis processes, and diagnostic results of sperm cells; the keyboard and mouse are used for users to input operation commands, such as selecting samples for analysis and adjusting analysis parameters; the printer is used to print diagnostic reports.

[0160] 2. Software Structure and Interaction Process

[0161] The user interface employs a graphical user interface (GUI) design, developed using the Qt framework. Users log in with their username and password to access the main system interface. On the main interface, users can import multimodal data from semen samples and view information such as data collection time and sample number. Clicking the "Analyze" button initiates the multimodal large language model analysis and diagnostic modules for processing. During processing, a progress bar and real-time analysis results (such as the current number of image frames processed and key indicators of feature extraction) are displayed on the interface. Upon completion of the analysis, a detailed analysis report of sperm cells is displayed, including values ​​for various quality indicators, comparisons with normal standards, and diagnostic suggestions. Users can zoom, save, and print the report using the keyboard and mouse, and can also view historical analysis records using the query function on the interface.

[0162] III. Action Process or Operation Steps

[0163] (I) Sample Preparation

[0164] Place the semen sample on the stage and use the three-dimensional motorized stage adjustment to center the sample in the microscope's field of view. Set the microscope magnification to 400x (initial setting, adjustable as needed), and turn on the CCD camera to prepare for acquiring image and video data.

[0165] (II) Multimodal Data Acquisition

[0166] Clicking the data acquisition button activates the microscopic imaging device, which begins acquiring microscopic images of sperm cells (10 frames per second) and dynamic motion videos (10 seconds in duration). Simultaneously, the sensor array collects real-time data on the temperature, pH value, and osmotic pressure of the semen sample. The text input device receives basic patient information and medical history data entered by the user.

[0167] The collected data is transmitted to the main control circuit board via the data interface. The main control circuit board temporarily stores the data in the cache chip and transmits the data to the preprocessing module via the Ethernet interface every second.

[0168] (III) Data Preprocessing

[0169] After receiving the image data, the high-performance computer in the preprocessing module immediately performs median filtering for noise reduction, with a processing time of 0.1 seconds per frame. Then, it performs histogram equalization for contrast enhancement, with a processing time of 0.2 seconds per frame.

[0170] For video data, the 10-second video is first broken down into 100 single-frame images (10 frames per second), and then each frame is preprocessed in the same way as the static images. The total processing time is 30 seconds (100 frames × 0.3 seconds / frame).

[0171] Upon receipt, the text data is immediately cleaned and standardized, with a processing time of 5 seconds.

[0172] After receiving the sensor data, it is first calibrated (corrected according to the sensor's calibration parameter table, processing time is 2 seconds), and then normalized (using the minimum-maximum normalization method, processing time is 1 second).

[0173] (iv) Multimodal Large Language Model Analysis

[0174] The preprocessed image features, text features, and sensor data features are input into their respective encoders for feature extraction. The image encoder takes 0.5 seconds per frame (100 frames × 0.5 seconds / frame = 50 seconds), the text encoder takes 10 seconds, and the sensor data encoder takes 5 seconds.

[0175] The feature vectors of the three modalities are concatenated and fused in the fusion layer to form a multimodal fused feature vector, with a processing time of 1 second.

[0176] The fused feature vectors are input into the decoder of the large language model for analysis. The decoder calculates through a self-attention mechanism and a feedforward neural network, and outputs the predicted results of indicators such as sperm concentration, motility, and normal morphology rate. The processing time is 30 seconds.

[0177] (V) Diagnosis

[0178] The inference engine, based on the prediction results output by the large language model, queries the standard rules in the diagnostic rule knowledge base and performs logical reasoning. First, it compares each indicator with WHO standards to determine whether it is normal (processing time: 10 seconds). Then, it analyzes the cause of the abnormality by combining medical history and sensor data (processing time: 15 seconds). Finally, it generates diagnostic suggestions (processing time: 5 seconds), with a total diagnosis time of 30 seconds.

[0179] (vi) Results Display and Interaction

[0180] After the diagnostic results are transmitted to the user interaction module, the analysis report is immediately displayed on the screen. Users can view detailed information, save the report, or print the report using the keyboard and mouse. The entire interaction process is real-time and has no noticeable delay.

[0181] Through the above specific implementation methods, this system can accurately and efficiently complete the multimodal data acquisition, preprocessing, analysis and diagnosis of sperm cells, providing reliable technical support for sperm quality assessment and male fertility diagnosis.

[0182] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A sperm cell analysis and diagnostic system based on a multimodal large language model, characterized in that, It includes a multimodal data collaborative processing unit, a cross-modal semantic alignment module, a dynamic diagnostic decision engine, and an adaptive evolutionary system: The multimodal data collaborative processing unit is used to simultaneously process four-dimensional input data including microscopic images, motion trajectory videos, biochemical test data, and medical record texts, and includes: A feature selector based on a gated attention mechanism is used to achieve dynamic weighted extraction of key features; A 3D CNN-Transformer hybrid architecture for modeling and analyzing temporal motion features; The cross-modal semantic alignment module maps heterogeneous data to a unified semantic space, and includes: A dual-tower contrastive learning framework achieves image-text alignment through visual and text encoding towers. A multi-scale feature pyramid was used to establish a correlation matrix between cell morphology features and clinical indicators. The dynamic diagnostic decision engine constructs a hierarchical reasoning mechanism with clinical interpretability, including a three-stage decision-making process: morphological abnormality detection → motor function assessment → comprehensive fertility index calculation. It develops a dynamic threshold adjustment module based on reinforcement learning, automatically optimizes diagnostic criteria according to contextual information such as patient age and medical history, realizes the visual traceability of diagnostic basis, locates abnormal cell regions through heat maps, and generates natural language reports that conform to clinical norms. The adaptive evolution system establishes a model update mechanism under a federated learning framework, supports the privacy and security sharing of multi-center medical data, designs a drift detection algorithm to monitor model performance degradation, triggers online incremental learning to maintain diagnostic accuracy, constructs a knowledge graph-driven abnormal case learning module, and mines potential feature patterns of rare diseases through graph neural networks.

2. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 1, characterized in that, The attention weight calculation of the feature selector satisfies: α_i=σ(W_q·[f_i||c]+b) Where f_i is the i-th feature vector, c is the clinical context vector, and W_q and b are learnable parameters; The 3D CNN-Transformer hybrid architecture includes: The SigLIP visual feature extraction model is used to extract contextual features from high-resolution images. InternVL 3.0 is a multimodal large language model that extracts multimodal and language capabilities from diverse multimodal data and plain text corpora.

3. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 1, characterized in that, The cross-modal semantic alignment module includes: The visual coding tower adopts a multi-task learning framework to simultaneously output sperm head morphology classification results and tail defect detection heatmaps. Text encoding tower extracts key values ​​of patients' reproductive history and seminal plasma biochemical indicators from medical records through medical entity recognition (NER); The loss function for contrastive learning is the improved NT-Xent loss: L_{cont}=-log[exp(sim(v,t) / τ) / (∑_{k≠t}exp(sim(v,k) / τ)] Where v represents visual features, t represents matching text features, and k represents negative samples.

4. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 1, characterized in that, The dynamic diagnostic decision engine includes: The three-stage reasoning submodule sequentially performs anomaly detection based on morphological features, vitality assessment combined with motion trajectory parameters, and calculation of a comprehensive fertility index integrating clinical history. The threshold dynamic adjustment submodule employs a reinforcement learning agent, whose state space is defined as follows: s_t = [age, abnormality rate, forward motor velocity, severity of medical history] The action space is ±10% of the diagnostic threshold.

5. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 4, characterized in that, In the three-stage reasoning submodule: The output of morphological anomaly detection includes the head cavitation ratio, acrosome integrity score, and tail bending angle statistics. Vitality assessment uses a weighted formula based on exercise parameters: Motility Score=0.6×VCL+0.3×ALH+0.1×BCF Where VCL is the curve velocity, ALH is the head lateral swing amplitude, and BCF is the whipping frequency.

6. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 1, characterized in that, The adaptive evolutionary system: The federated learning framework employs a gradient aggregation protocol based on homomorphic encryption; Knowledge graphs contain three types of entity relations: (Types of sperm abnormalities) - [requires] -> (ICSI treatment); (Oxidative stress) - [affects] -> (DNA fragmentation rate); The drift detection module monitors changes in the distribution of input data using KL divergence.

7. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 1, characterized in that, Also includes: The report generation module automatically outputs diagnostic reports that conform to WHO standards, including heatmaps of abnormal cell locations, animations reconstructing their movement trajectories, and treatment recommendations described in natural language.

8. The sperm cell analysis and diagnosis system based on a multimodal large language model according to claim 1, characterized in that, It also includes a sperm cell analysis method using the system described in any one of claims 1-7, comprising the following steps: 1) Extract depth features from microscopic images and motion videos using a multimodal data collaborative processing unit; 2) Establish the association between image features and medical record text using a cross-modal semantic alignment module; 3) The dynamic diagnostic decision engine generates hierarchical diagnostic conclusions; 4) Update model parameters through an adaptive evolutionary system.

9. A computer-readable storage medium storing program instructions, characterized in that, When the instructions are executed by the processor, all the steps of the method of claim 8 are implemented.

Images