Cornea image death time prediction method based on multi-modal Transform-diffusion model

Abstract

The invention provides a cornea image death time prediction method based on a multi-modal Transform-diffusion model, and belongs to the technical field of forensic science, and the method comprises the steps: firstly obtaining cornea image data and environmental parameter data; preprocessing the cornea image data; image quality enhancement is carried out by using a diffusion model, and the noise problem in a low-quality image is effectively solved; carrying out feature extraction on the enhanced cornea image through a VisionTransform, and capturing a complex nonlinear relationship; multi-modal fusion is carried out on cornea image features and environmental parameter data, and various influence factors are comprehensively considered; finally, a death time prediction result is generated based on the fusion features, the method further comprises the steps that the model is compressed and deployed to lightweight equipment, field application is achieved, and the method achieves high-accuracy prediction within the range of 0-72 hours and particularly has obvious advantages in long-time prediction over 24 hours.

Description

Technical Field

[0001] The present invention relates to the field of forensic science and technology, and specifically to a method for predicting time of death from corneal images based on a multimodal Transformer-diffusion model, and in particular to analyzing corneal images using deep learning technology to achieve high-precision prediction of time of death. Background Art

[0002] In forensic practice, accurately determining the time of death is crucial for criminal investigations. Traditional methods for determining the time of death include body temperature, gastric content analysis, and ocular chemistry. Body temperature is significantly affected by ambient temperature, gastric content analysis is significantly affected by individual differences, and ocular chemistry can damage the cornea and requires complex procedures. These methods generally suffer from low accuracy, sensitivity to environmental factors, and complex procedures.

[0003] With the development of computer vision and artificial intelligence technologies, methods for estimating the time of death based on image analysis have gradually emerged. The cornea, as a transparent tissue exposed to the outside world, undergoes significant changes in opacity over time after death. This change exhibits a certain regularity, providing the potential for predicting time of death through image analysis. However, existing methods for predicting time of death based on corneal images primarily rely on simple image processing techniques and traditional machine learning algorithms, which have difficulty capturing complex nonlinear relationships and have limited processing capabilities for low-quality images. Prediction accuracy is particularly reduced under conditions such as high image noise and uneven lighting.

[0004] Furthermore, existing technologies also have shortcomings in data collection and processing. The lack of standardized, high-quality corneal image databases, particularly those spanning more than 24 hours under diverse environmental conditions, limits the effective training of deep learning models. Furthermore, existing methods often focus solely on the corneal image itself, ignoring the impact of environmental parameters (such as temperature and humidity) on corneal changes, limiting the accuracy and reliability of prediction results. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a corneal image time-to-death prediction method based on a multimodal Transformer-diffusion model, which can overcome the limitations of existing methods in processing complex nonlinear relationships and low-quality images, and improve the accuracy and reliability of time-to-death prediction.

[0006] The present invention proposes a corneal image time-to-death prediction method based on a multimodal Transformer-diffusion model, comprising: Acquiring corneal image data and environmental parameter data; Preprocessing the corneal image data to obtain a preprocessed corneal image; Based on the preprocessed corneal image, performing image quality enhancement using a diffusion model to obtain an enhanced corneal image; Using VisionTransformer to perform feature extraction on the enhanced corneal image to obtain corneal image features; Performing multimodal fusion on the corneal image features and the environmental parameter data to obtain fused features; Based on the fused features, a death time prediction result is generated.

[0007] Preferably, the acquiring of corneal image data and environmental parameter data comprises: Collect corneal images and record environmental parameters such as temperature and humidity at the time of image capture; Scoring the clarity, illumination uniformity, corneal integrity, focus accuracy, shooting angle, and environmental interference of the corneal image to obtain an image quality score; Based on the image quality score, qualified corneal image data are screened.

[0008] Preferably, the method for scoring the clarity, illumination uniformity, corneal integrity, focus accuracy, shooting angle and environmental interference of the corneal image to obtain the image quality score is: Each dimension is scored from 1 to 5 points; When the total score is greater than or equal to 25 points, it is judged as a good image; When the total score is between 20 and 24, the image is determined to require review; When the total score is less than 20 points or the score of any dimension is less than or equal to 2 points, the image is judged as unqualified.

[0009] Preferably, the preprocessing of the corneal image data to obtain a preprocessed corneal image comprises: performing corneal region segmentation on the corneal image; Adjust the segmented corneal region image to a uniform size; Normalize pixel values; Perform image enhancement, including contrast adaptive adjustment, illumination equalization and edge enhancement.

[0010] Preferably, the image quality enhancement using the diffusion model includes: Design a forward noise adding process to gradually add noise to the original corneal image; The diffusion model is trained to learn the reverse denoising process from the noisy image to the original image; The trained diffusion model is used to denoise and repair the low-quality corneal image to obtain the enhanced corneal image; Generate diverse corneal image samples to expand the training dataset.

[0011] Preferably, the extracting features from the enhanced corneal image using VisionTransformer comprises: Segmenting the enhanced corneal image into a plurality of image blocks; Performing linear projection on the image block and adding position encoding information to obtain an embedded representation of the image block; Input the image block embedding representation into the Transformer encoder; Through the multi-head self-attention mechanism and feedforward neural network, the global relationship between image blocks is captured to obtain corneal image features.

[0012] Preferably, the multimodal fusion of the corneal image features and the environmental parameter data to obtain fused features comprises: Converting the environmental parameter data into a parameter embedding vector; Design a feature fusion module that combines corneal image features and parameter embedding vectors; The attention mechanism is used to dynamically adjust the weights of different modal features to obtain fused features.

[0013] Preferably, generating a death time prediction result based on the fusion feature includes: Design a classification head module to map the fused features to the probability distribution of death time intervals; One-hot encoding is used to represent the death time interval, and the number of intervals is 12, corresponding to different death time periods; Calculate the confidence score of the prediction result; Low-confidence predictions are flagged to indicate that manual review may be required.

[0014] As an option, it also includes: Jointly optimize the Transformer model and the diffusion model to improve overall system performance; Design a feedback mechanism so that Transformer prediction errors guide the improvement of the diffusion model; Implement a serial processing flow, with the diffusion model as the pre-processing engine and the Transformer model as the main prediction engine.

[0015] Preferably, the method further includes a step of deploying the method on a lightweight device, including: Compress Transformer and diffusion models, including knowledge distillation, parameter quantization, and model pruning; Convert the compressed model into a format suitable for edge devices; A full-process system for data acquisition, processing and result display is implemented on an integrated hardware platform; The integrated hardware platform includes a high-resolution camera module, a lightweight computing unit, a multi-parameter sensor, a display screen, and an integrated battery power supply system.

[0016] The beneficial effects of the present invention include:

[0017] 1. By establishing a multimodal fusion framework and combining corneal image data and environmental parameter data, we can comprehensively capture various factors affecting corneal changes and improve the accuracy and robustness of the prediction model.

[0018] 2. Use the diffusion model to enhance image quality, effectively deal with the noise problem in low-quality corneal images, improve image clarity, and provide a better data foundation for subsequent analysis.

[0019] 3. Leverage VisionTransformer’s powerful feature extraction capabilities to capture complex patterns and nonlinear relationships in corneal images, enabling more accurate time-to-death prediction.

[0020] 4. Establish a complete process of shooting - importing analysis - presenting time of death, simplify the operation steps and improve the efficiency of forensic work.

[0021] 5. Through model lightweight technology, it can be deployed on portable devices to meet the needs of on-site forensic work. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an overall flow chart of the corneal image time-to-death prediction method based on the multimodal Transformer-diffusion model of the present invention.

[0023] Figure 2 Schematic diagram of the corneal image quality assessment system of the present invention.

[0024] Figure 3 Schematic diagram of the diffusion model image quality enhancement process of the present invention.

[0025] Figure 4 Schematic diagram of the VisionTransformer structure of the present invention.

[0026] Figure 5 Schematic diagram of the structure of the multimodal feature fusion module of the present invention. DETAILED DESCRIPTION

[0027] Please refer to Figure 1 - Figure 5 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0028] Reference Figure 1 The present invention provides a method for predicting time of death of corneal images based on a multimodal Transformer-diffusion model, which mainly includes the following steps:

[0029] First, obtain corneal image data and environmental parameter data. In practical applications, a high-resolution camera (e.g., a Canon EOS M650) equipped with a polarizer can be used to capture corneal images while simultaneously recording ambient temperature, humidity, and other parameters. Preferably, the ambient temperature is controlled within the range of 18°C ​​to 26°C, as corneal changes are more regular within this temperature range, which helps improve prediction accuracy.

[0030] Next, the acquired corneal image data is preprocessed to obtain a preprocessed corneal image. This preprocessing step includes corneal region segmentation, image resizing, pixel value normalization, and preliminary image enhancement. In one embodiment of the present invention, the image can be resized to a uniform size of 224×224 pixels and pixel value normalization is performed to bring the pixel values ​​into a uniform range of 0-1. This helps improve the stability and accuracy of subsequent processing.

[0031] The diffusion model is then used to enhance the preprocessed corneal image quality, resulting in an enhanced corneal image. The diffusion model effectively removes noise and improves image quality through a forward denoising and backward denoising process. Furthermore, the diffusion model generates diverse training samples, enhancing the model's generalization and robustness.

[0032] Next, VisionTransformer is used to extract features from the enhanced corneal image to obtain corneal image features. VisionTransformer first segments the image into multiple image patches and then uses a multi-head self-attention mechanism to capture the global relationship between the image patches, thereby extracting deep features of the corneal image.

[0033] Subsequently, the corneal image features are fused with the environmental parameter data through multimodal fusion to obtain fused features. Multimodal fusion dynamically adjusts the weights of different modal features through the attention mechanism to achieve effective information integration.

[0034] Finally, based on the fused features, a time-of-death prediction is generated. The prediction results can be one-hot encoded to represent different time-of-death intervals. The prediction confidence is also calculated, and low-confidence predictions are flagged, indicating that manual review may be required.

[0035] In the present invention, the quality of corneal images has a significant impact on the prediction results. Therefore, the present invention establishes a strict image quality evaluation system.

[0036] Reference Figure 2 The present invention performs multi-dimensional scoring on the acquired corneal images, including six dimensions: image clarity, illumination uniformity, corneal integrity, focus accuracy, shooting angle, and environmental interference, and each dimension is scored on a scale of 1 to 5 points.

[0037] For image clarity, a score of 5 indicates that the texture is clearly visible and the corneal fine structure is visible; a score of 3 indicates mild blurring but the edges are still clear; and a score of 1 indicates severe blurring and inability to discern corneal texture. Preferably, the image clarity score is no less than 3 points to ensure that the corneal texture information in the image can be effectively extracted.

[0038] For illumination uniformity, a score of 5 indicates no reflection and uniform illumination, a score of 3 indicates partial reflection but less than 20% of the cornea, and a score of 1 indicates strong reflection or shadow covering more than 50% of the cornea. Preferably, an illumination uniformity score of 3 or higher helps minimize interference caused by uneven illumination.

[0039] For corneal integrity, a score of 5 indicates complete and undamaged cornea, a score of 3 indicates minor damage but less than 10% of the cornea, and a score of 1 indicates a corneal tear or loss exceeding 30%. Preferably, the corneal integrity score should be no less than 4 to ensure the integrity of corneal morphological information.

[0040] For focus accuracy, a score of 5 indicates that the entire cornea is in focus, a score of 3 indicates that a portion of the cornea is in focus, and a score of 1 indicates that the iris texture is blurred due to lack of focus. Preferably, a focus accuracy score of 4 or higher ensures the overall clarity of the image.

[0041] For the shooting angle, 5 points means vertical to the center of the cornea with an error of less than 5 degrees, 3 points means tilted 10-20 degrees, and 1 point means tilted more than 30 degrees. Preferably, the shooting angle score is not less than 4 points, which reduces the deformation caused by the change of viewing angle.

[0042] For environmental interference, 5 points means no contamination, 3 points means slight contamination but can be corrected by software, and 1 point means obvious contamination such as blood stains or dust. Preferably, the environmental interference score is not less than 4 points, which reduces the impact of external factors on image quality.

[0043] Based on the scoring of the above six dimensions, the present invention uses the following criteria for image screening:

[0044] When the total score is greater than or equal to 25 points, it is judged as a good image and directly enters the analysis process;

[0045] When the total score is between 20 and 24, the image is judged to require review, and a third party will decide whether to use it;

[0046] When the total score is less than 20 points or the score of any dimension is less than or equal to 2 points, the image is judged as unqualified and directly eliminated.

[0047] Through this strict image quality evaluation system, the present invention ensures that the corneal images entering the analysis process have sufficiently high quality, providing a good data basis for subsequent processing.

[0048] After obtaining a qualified corneal image, a series of preprocessing operations are required to prepare for subsequent analysis. The corneal image preprocessing of the present invention mainly includes four steps:

[0049] First, the corneal region is segmented from the corneal image. Because the original image may contain non-corneal regions such as eyelids and eyelashes, image segmentation techniques are required to extract the corneal region. In one embodiment of the present invention, a U-Net-based semantic segmentation method can be used to identify and extract the corneal region. Preferably, the segmentation accuracy (IoU, Intersection over Union) is no less than 0.9, ensuring accurate extraction of the corneal region.

[0050] Secondly, the segmented corneal region images are resized to a uniform size. Considering the input requirements of the subsequent VisionTransformer model, the present invention resizes the corneal images to a uniform size of 224×224 pixels. During this resizing process, a bicubic interpolation algorithm is used to preserve image detail.

[0051] Then, pixel value normalization is performed. Normalizing the pixel values ​​of the three RGB channels to the range of 0-1 helps improve the stability of model training. The normalization formula is:

[0052] ,

[0053] in is the normalized pixel value, is the minimum value of the channel (usually 0), The maximum value of the channel (usually 255).

[0054] Finally, image enhancement is performed. This invention employs multiple image enhancement techniques, including contrast adaptive adjustment, illumination equalization, and edge enhancement. Contrast adaptive adjustment uses the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm, which divides the image into several subregions (e.g., an 8×8 grid), performs histogram equalization on each subregion, and then performs bilinear interpolation, effectively improving the local contrast of the image. Illumination equalization removes the effects of uneven illumination through Gaussian filtering and subtraction operations. Edge enhancement uses a nonlinear edge enhancement filter to enhance corneal edge and texture information.

[0055] Through the above preprocessing steps, the present invention effectively improves the quality and standardization of corneal images, laying the foundation for subsequent deep learning analysis.

[0056] The pre-processed corneal image may still have some noise and quality issues, especially when the image is acquired under non-ideal shooting conditions. To this end, the present invention introduces a diffusion model to further enhance the image quality.

[0057] The diffusion model is a generative model that generates high-quality images by gradually adding noise and learning the denoising process. The diffusion model of the present invention includes two key parts: the forward denoising process and the reverse denoising process.

[0058] The forward noise addition process is designed as follows:

[0059] From the original corneal image Start by Step by step, Gaussian noise is gradually added to obtain a series of noisy images. The noise addition at each step can be expressed as:

[0060] ,

[0061] in: is the noise image at step t, is the noise image at step t-1, is the noise scheduling parameter (usually set as a decreasing sequence ranging from 0.999 to 0.9), is standard Gaussian noise, which conforms to the normal distribution with mean 0 and variance 1 , is the identity matrix. Preferably, the noise step number T is set to 1000, which provides a sufficiently smooth noise addition process.

[0062] The reverse denoising process is the process of learning to restore the original image from the noisy image. Predict the noise added at each step and then restore the original image through reverse reasoning. The training goal is to minimize the following loss function:

[0063] ,

[0064] in: is the loss function, are neural network parameters, is the actual noise added, is the noise predicted by the network, is the time step, Indicates the expected value, Represents the square of the Euclidean norm.

[0065] After training is completed, the reverse denoising process can be expressed as:

[0066] ,

[0067] in: To restore the lower noise image, is the cumulative noise parameter, represents all the values ​​from i=1 to t The product of A small noise is added at each step (usually set to ), is standard Gaussian noise, used to increase sampling diversity.

[0068] The diffusion model of this invention is not only used to enhance image quality but also to generate diverse training samples, expanding the training dataset. In particular, for time periods with scarce data (such as corneal images older than 24 hours), conditional generation techniques can be used to synthesize more samples, improving the model's prediction accuracy during these time periods.

[0069] In practical applications, the diffusion model's denoising performance outperforms traditional methods. For example, for corneal images with 80% random noise added, the proposed diffusion model achieves a peak signal-to-noise ratio (PSNR) of 31.2 dB, significantly higher than traditional denoising methods such as APGL (26.63 dB), TNNR (27.47 dB), and DNN (28.79 dB).

[0070] The corneal image enhanced by the diffusion model is fed into VisionTransformer (ViT) for feature extraction. ViT is a deep learning model that applies the Transformer architecture to computer vision tasks and can effectively capture global dependencies in images.

[0071] The VisionTransformer feature extraction process of the present invention is as follows:

[0072] First, the enhanced corneal image (size 224×224 pixels) is segmented into multiple 16×16 pixel patches, resulting in 196 patches. Each patch is converted into an embedding vector with a dimension of 768 by linear projection. The patch embedding representation can be expressed as:

[0073] ,

[0074] in: is the initial embedding representation (dimension is 197×768), is a special classification tag (a learnable parameter vector with a dimension of 768), is the first image block (dimension is 256, i.e. 16×16 pixels), is the embedding matrix (dimension is 256×768, mapping the 256-dimensional image block to the 768-dimensional feature space), is the position encoding (a learnable parameter matrix with a dimension of 197×768, used to preserve spatial position information), and the semicolon “;” represents a vector concatenation operation.

[0075] The image patch embedding representation is then fed into the Transformer encoder. The Transformer encoder consists of multiple layers, each of which includes a multi-head self-attention mechanism and a feed-forward neural network. The multi-head self-attention mechanism allows the model to simultaneously focus on information from different regions of the image, capturing the complex relationships between image patches. The self-attention calculation formula is:

[0076] ,

[0077] in: is the query matrix (Query, dimension is n× , n is the sequence length), is the key matrix (Key, dimension is n× ), is the value matrix (Value, dimension is n×d_v), is the dimension of the key, represents the transposed matrix of K, softmax is the softmax activation function, which converts the input into a probability distribution. It is a scaling factor to prevent the softmax gradient from disappearing due to the dot product value being too large.

[0078] In multi-head attention, the attention mechanism is run in parallel h times (h=8 in this paper), and then the results are spliced ​​together:

[0079] ,

[0080] Among them: head Attention (dimension is d_modelxd_k), (dimension is d_modelxd_k), (of dimension d_model x d_v) and (with dimensions of hd_v x d_model) is a learnable parameter matrix, Concat represents a concatenation operation that concatenates the outputs of h heads, and d_model is the model dimension (768 in this invention).

[0081] In the feedforward neural network part, a two-layer fully connected network is used for nonlinear transformation:

[0082] ,

[0083] Among them: GELU is the Gaussian error linear unit activation function, 、 、 、 is a learnable parameter. Preferably, the hidden layer dimension of the feedforward network is set to 3072, which provides sufficient model capacity.

[0084] After processing by L layers of Transformer encoder (L=12 in this invention), the deep feature representation of the corneal image is finally obtained. In particular, the classification mark The corresponding output is used as the global image feature for subsequent death time prediction.

[0085] The Vision Transformer of this invention has the following advantages over traditional convolutional neural networks (CNNs):

[0086] 1. Capable of capturing long-range dependencies in corneal images and identifying subtle patterns of corneal changes over time;

[0087] 2. The attention mechanism provides interpretability of model decisions, helping to understand the image regions that the model focuses on;

[0088] 3. It is more robust to image deformation and position changes and can adapt to corneal images under different shooting conditions.

[0089] Corneal changes are influenced by multiple factors, and relying solely on image features may not fully capture the relationship between time of death and corneal condition. Therefore, this paper introduces multimodal fusion technology, fusing corneal image features with environmental parameter data (such as temperature and humidity) to improve the accuracy and robustness of predictions.

[0090] The multimodal feature fusion process is as follows:

[0091] First, the environmental parameter data is converted into a parameter embedding vector. Assuming that the environmental parameters contain m dimensions (such as temperature, humidity, etc.), each parameter is first normalized and then converted into an embedding vector of dimension d through a linear layer:

[0092] ,

[0093] in: is the parameter embedding vector (dimension is d, d=768 in this invention), is the normalized environmental parameter vector (dimension is m), is the weight matrix (dimension is m×d), is the bias vector (dimension d).

[0094] Then, a feature fusion module is designed to combine corneal image features and parameter embedding vectors. The present invention adopts a fusion method based on the attention mechanism to dynamically adjust the weights of different modal features. The fusion process can be expressed as:

[0095] ,

[0096] in: is the fusion feature (dimension is d), Image features extracted by Vision Transformer (dimension d), is the parameter embedding vector (dimension d), and is the attention weight, satisfying .

[0097] The attention weights are calculated using a learnable attention network:

[0098] ,

[0099] in: is the weight matrix (dimension is 2×2d), is the bias vector (dimension is 2), It means that the image features and parameter features are concatenated into a vector (dimension is 2d), and the softmax function ensures that the sum of the two output weights is 1.

[0100] In one embodiment of the present invention, a cross-modal attention mechanism is further introduced to allow for the interaction between features from different modalities. The attention of image features to parameter features and the attention of parameter features to image features are calculated separately and then fused, which enhances the model's ability to capture complex relationships between modalities.

[0101] A key advantage of multimodal fusion is its improved adaptability to varying environmental conditions. For example, corneal opacification typically accelerates at higher temperatures (above 26°C), while it progresses more slowly at lower temperatures (below 18°C). By integrating environmental parameter information, the model can automatically adjust its interpretation of image features, improving prediction accuracy.

[0102] Based on the fused features, the present invention designs a classification head module to generate the death time prediction results. The classification head module maps the fused features to the probability distribution of the death time interval and uses one-hot encoding to represent different death time intervals.

[0103] Specifically, the classification head module contains a multi-layer perceptron (MLP) that maps the fused features $f_{fused}$ to k output units (k=12 in this invention, corresponding to 12 death time intervals):

[0104] ,

[0105] in: is the predicted probability distribution of death time interval (dimension is k), is the first layer weight matrix (dimension is d×d_h, d_h is the hidden layer dimension, set to 1024), is the first layer bias vector (dimension d_h), is the second layer weight matrix (dimension is d_h×k), is the second layer bias vector (dimension is k), GELU is the Gaussian error linear unit activation function, and the softmax function converts the output into a probability distribution.

[0106] The 12 time intervals of death are divided into the following categories: 0-6 hours, 6-12 hours, 12-18 hours, 18-24 hours, 24-30 hours, 30-36 hours, 36-42 hours, 42-48 hours, 48-54 hours, 54-60 hours, 60-66 hours, and 66-72 hours. This division maintains sufficient accuracy while taking into account the needs of actual forensic work.

[0107] For the confidence evaluation of the prediction results, the present invention calculates the maximum value of the prediction probability distribution as the confidence score:

[0108] ,

[0109] in: is the confidence score, is the predicted probability distribution, The function returns the maximum value in a vector.

[0110] When the confidence score falls below a threshold (e.g., 0.75), the system marks the prediction as low confidence, indicating that manual review may be required. This mechanism increases the reliability of the prediction results, especially when dealing with edge cases or atypical corneal images.

[0111] During model training, the cross entropy loss function is used:

[0112] ,

[0113] in: is the loss value, is the true label (one-hot encoding, dimension is k), is the predicted probability of the i-th time interval, is the natural logarithm function, represents the sum of k time intervals.

[0114] In order to deal with the problem of unbalanced number of samples in different time intervals, this paper introduces weighted cross entropy loss to assign higher weights to time intervals with fewer samples:

[0115] ,

[0116] in: is the weighted loss value, is the weight of the i-th time interval, which is inversely proportional to the number of samples in the interval. The calculation formula is: , is the total number of samples, is the number of samples in the i-th interval.

[0117] Experimental validation demonstrated that the proposed method for predicting time of death achieved 99% accuracy in a 72-hour dog study, significantly outperforming traditional methods. In particular, the method performed exceptionally well in predictions over extended periods of time, exceeding 24 hours. This holds significant value for expanding the scope of forensic time of death assessment.

[0118] In order to further improve the system performance, the present invention jointly optimizes the Transformer model and the diffusion model and designs a collaborative working mechanism.

[0119] The joint optimization process adopts a two-stage training strategy: first pre-train the Transformer model and the diffusion model separately, and then perform end-to-end joint fine-tuning. The loss function of the joint fine-tuning stage consists of two parts:

[0120] ,

[0121] in: For joint losses, is the classification loss (for death time prediction), is the diffusion model loss (for image enhancement), and The joint optimization method enables the two models to coordinate with each other and improve the overall performance.

[0122] In practical applications, the Transformer and Diffusion models use a serial processing flow: the Diffusion model serves as the pre-processing engine, responsible for image quality enhancement; the Transformer model serves as the main prediction engine, responsible for feature extraction and temporal prediction. This serial architecture simplifies data flow and reduces system complexity.

[0123] Furthermore, the present invention incorporates a feedback mechanism that allows Transformer prediction errors to guide improvements in the diffusion model. Specifically, when the Transformer's prediction accuracy for a certain type of corneal image is low, the system automatically flags these images and increases the diffusion model's processing weight for these images in subsequent training, improving the quality enhancement of these images.

[0124] Experimental results show that this collaborative optimization strategy can significantly improve system performance. Compared with using the Transformer or diffusion model alone, the jointly optimized system is more stable in low-quality image processing and boundary case prediction, with an average accuracy improvement of 3.5%.

[0125] To meet the needs of on-site forensic work, the present invention deploys the method on lightweight equipment to achieve portable operation.

[0126] Model compression is a key step in lightweight deployment. This paper uses techniques such as knowledge distillation, parameter quantization, and model pruning to compress the Transformer model and the diffusion model.

[0127] In the knowledge distillation process, a large teacher model is first trained, and then the output of the teacher model is used to guide the training of a small student model. The distillation loss function is:

[0128] ,

[0129] in is the distillation loss, is the cross entropy loss (the loss between the student model output and the true label), is the KL divergence loss (the difference between the student model output and the teacher model output), is the true label, is the output of the student model, is the output of the teacher model, is the balance coefficient (set to 0.5 in the present invention), and KL divergence (Kullback-Leibler divergence) is used to measure the difference between two probability distributions.

[0130] Parameter quantization converts 32-bit floating-point parameters into 8-bit integer representations, significantly reducing model size and computational complexity. Pruning technology removes network connections that contribute less to predictions, further reducing the model burden. Through these techniques, the present invention reduces model size by over 80% while maintaining at least 97% of the original accuracy.

[0131] The lightweight model is preferably converted to ONNX format and further optimized using TensorRT to fully utilize the computing power of edge devices. This reduces inference time from several seconds to less than 1 second, meeting real-time processing requirements.

[0132] The integrated hardware platform includes a high-resolution camera module (Canon EOS M650), a lightweight computing unit (NVIDIA Jetson TX2), multi-parameter sensors, a 7-inch touchscreen display, and an integrated battery power system. The entire system weighs less than 1.5 kilograms and has a battery life of over 8 hours, making it suitable for on-site forensic work environments.

[0133] In terms of software, the present invention has developed an intuitive user interface to simplify the operation process. The user simply takes a corneal image, and the system automatically performs preprocessing, quality enhancement, and time of death prediction, and generates a detailed report. The report includes the predicted time of death interval, confidence score, and key corneal feature markers, making it easier for forensic personnel to understand and use the prediction results.

[0134] This paper provides a method for predicting time of death from corneal images based on a multimodal Transformer-diffusion model. By combining advanced deep learning techniques with multimodal data fusion strategies, it effectively improves the accuracy and reliability of time of death prediction. Furthermore, through model lightweighting and hardware integration, it enables deployment on portable devices, meeting the needs of on-site forensic work. This method performs well within the 0-72 hour range, particularly for long-term predictions exceeding 24 hours, providing a new solution for technological advancement and practical application in the field of forensic science.

[0135] The above-described embodiments merely represent specific implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

Claims

1. A corneal image time-to-death prediction method based on a multimodal Transformer-diffusion model, characterized by: include: Acquiring corneal image data and environmental parameter data; Preprocessing the corneal image data to obtain a preprocessed corneal image; Based on the preprocessed corneal image, performing image quality enhancement using a diffusion model to obtain an enhanced corneal image; Using VisionTransformer to perform feature extraction on the enhanced corneal image to obtain corneal image features; Performing multimodal fusion on the corneal image features and the environmental parameter data to obtain fused features; Based on the fused features, a death time prediction result is generated.

2. The method according to claim 1, characterized in that The acquisition of corneal image data and environmental parameter data includes: Collect corneal images and record the temperature and humidity environmental parameters at the time of image capture; Scoring the clarity, illumination uniformity, corneal integrity, focus accuracy, shooting angle, and environmental interference of the corneal image to obtain an image quality score; Based on the image quality score, qualified corneal image data are screened.

3. The method according to claim 2, characterized in that The method for scoring the clarity, illumination uniformity, corneal integrity, focus accuracy, shooting angle and environmental interference of the corneal image to obtain the image quality score is: Each dimension is scored from 1 to 5 points; When the total score is greater than or equal to 25 points, it is judged as a good image; When the total score is between 20 and 24, the image is determined to require review; When the total score is less than 20 points or the score of any dimension is less than or equal to 2 points, the image is judged as unqualified.

4. The method according to claim 1, wherein The preprocessing of the corneal image data to obtain the preprocessed corneal image comprises: performing corneal region segmentation on the corneal image; Adjust the segmented corneal region image to a uniform size; Normalize pixel values; Perform image enhancement, including contrast adaptive adjustment, illumination equalization and edge enhancement.

5. The method according to claim 1, wherein The image quality enhancement using the diffusion model includes: Design a forward noise adding process to gradually add noise to the original corneal image; The diffusion model is trained to learn the reverse denoising process from the noisy image to the original image; The trained diffusion model is used to denoise and repair the low-quality corneal image to obtain the enhanced corneal image; Generate diverse corneal image samples to expand the training dataset.

6. The method according to claim 1, characterized in that The feature extraction of the enhanced corneal image using VisionTransformer includes: Segmenting the enhanced corneal image into a plurality of image blocks; Performing linear projection on the image block and adding position encoding information to obtain an embedded representation of the image block; Input the image block embedding representation into the Transformer encoder; Through the multi-head self-attention mechanism and feedforward neural network, the global relationship between image blocks is captured to obtain corneal image features.

7. The method according to claim 1, characterized in that The multimodal fusion of the corneal image features and the environmental parameter data to obtain fused features includes: Converting the environmental parameter data into a parameter embedding vector; Design a feature fusion module that combines corneal image features and parameter embedding vectors; The attention mechanism is used to dynamically adjust the weights of different modal features to obtain fused features.

8. The method according to claim 1, characterized in that Generating a death time prediction result based on the fusion feature includes: Design a classification head module to map the fused features to the probability distribution of death time intervals; One-hot encoding is used to represent the death time interval, and the number of intervals is 12, corresponding to different death time periods; Calculate the confidence score of the prediction result; Low-confidence predictions are marked to indicate that manual review is required.

9. The method according to any one of claims 1 to 8, characterized in that: Also includes: Jointly optimize the Transformer model and the diffusion model to improve overall system performance; Design a feedback mechanism so that Transformer prediction errors guide the improvement of the diffusion model; Implement a serial processing flow, with the diffusion model as the pre-processing engine and the Transformer model as the main prediction engine.

10. The method according to claim 1, characterized in that The method is also deployed on a lightweight device, including: Compress Transformer and diffusion models, including knowledge distillation, parameter quantization, and model pruning; Convert the compressed model into a format suitable for edge devices; A full-process system for data acquisition, processing and result display is implemented on an integrated hardware platform; The integrated hardware platform includes a high-resolution camera module, a lightweight computing unit, a multi-parameter sensor, a display screen, and an integrated battery power supply system.

Images