A Method for Constructing and Applying a Multi-Temporal Image Risk Classification Model

Abstract

This paper presents a method for constructing a multi-temporal image risk classification model, belonging to the field of intelligent image analysis. The method involves standardizing preprocessing, multimodal registration, and longitudinal registration of the image to extract target regions, anomalous regions, historical anomalous regions, locally changed regions, and surrounding extended regions, and constructing longitudinal difference maps, regional change rates, and time interval codes. Then, a progressive multi-stage risk classification model is constructed, including a basic image representation learning module, a longitudinal feature fusion module, an auxiliary variable encoding module, a first-stage coarse-grained risk classifier, a second-stage fine-grained risk classifier, an uncertainty assessment module, and a difficult sample re-discrimination module. The model is optimized using a progressive training strategy, moving from easy to difficult, from single-timepoint to multi-timepoint, and from coarse to fine. Fine-grained classification and difficult sample re-discrimination are performed on non-low-risk, suspected anomalous, anomalous extension-related, or uncertain samples. The multi-stage outputs are then fused to obtain information such as risk categories and probabilities for each category.

Description

Technical Field

[0001] This invention relates to the fields of intelligent image analysis, deep learning model training, multi-temporal image processing, risk classification model construction, and uncertainty assessment, specifically to a method for constructing a multi-temporal image risk classification model executed by a computer.

[0002] This invention is applicable to the automated processing of multi-temporal images of target objects and the construction of risk classification models. It can utilize one or more image modalities from visible light images, infrared thermal images, X-ray images, ultrasonic images, depth maps, 3D point cloud images, or other detection images, and can combine auxiliary attribute information such as object attributes, material parameters, operating conditions, environmental parameters, maintenance records, detection time intervals, and subsequent status labels to output risk category probabilities, model confidence, uncertainty scores, key image regions, and longitudinal change prompts.

[0003] The risk classification results described in this invention are computer-output image risk classification information. They do not directly limit specific engineering treatment measures, but are used as a reference for manual review, status assessment, maintenance priority determination, or subsequent inspection. The following explanation uses multi-temporal image risk classification of industrial components as an example, but this does not constitute a limitation on the scope of protection of this invention. Background Technology

[0004] In the condition assessment of target objects such as industrial components, mechanical parts, building structures, and transportation facilities, these objects may exhibit phenomena such as cracks, corrosion, wear, spalling, thermal anomalies, abnormal deformation, or changes in historical maintenance areas during long-term service. Multi-temporal images can record the condition changes of the same target object at different inspection points, serving as an important data source for risk classification and condition evolution analysis.

[0005] However, multi-temporal image risk classification tasks typically face challenges such as multiple image modalities, inconsistent detection time points, complex local anomaly morphological changes, and the similarity between transient interference and real anomaly propagation. Taking industrial components as an example, surface stains, light reflection, maintenance marks, localized corrosion, and real crack propagation may have similar appearances in images, making it difficult for single-time-point images or single-stage classification models to stably distinguish them.

[0006] Existing image risk classification methods typically input single-time-point images or simply stitched multimodal images directly into deep neural networks, with a single classifier outputting the final category. While these methods can learn image features, they generally suffer from the following shortcomings: First, insufficient utilization of longitudinal information; some methods use only single-time-point images or simply stitch together images from multiple time points, failing to explicitly model the dynamic changes between the baseline time point, early detection time points, and subsequent detection time points. Second, significant class imbalance; the number of normal or low-risk samples is usually greater than that of anomalous expansion, minority class high-risk samples, or complex boundary samples, making the model prone to shifting towards the majority class. Third, single-stage classifiers lack a coarse-to-fine and easy-to-difficult training and inference mechanism, resulting in a high misclassification rate on boundary samples and uncertain samples. Fourth, existing outputs often only provide category labels, lacking interpretable key image regions, longitudinal change cues, and model uncertainty information, which is detrimental to manual review.

[0007] Therefore, a model construction method is needed for multi-temporal image risk classification scenarios. This method should be able to perform standardized preprocessing and longitudinal registration on multimodal, multi-time-point images, extracting target regions, anomalous regions, historical anomalous regions, locally changed regions, and surrounding extended regions to construct a longitudinal difference map. Through a progressive multi-stage learning strategy, the model should first learn typical samples and coarse-grained risk distinctions, then gradually learn difficult samples, minority class samples, and fine-grained risk categories. Simultaneously, it should be able to identify inter-stage inconsistencies or low-confidence samples through uncertainty scoring and trigger re-discrimination of difficult samples, thereby improving the stability and interpretability of multi-temporal image risk classification. Summary of the Invention

[0008] The purpose of this invention is to provide a computer-executed method for constructing a multi-temporal image risk classification model, which addresses the problems of existing multi-temporal image risk classification models, such as insufficient utilization of longitudinal image changes, weak ability to distinguish between transient anomalies and true anomalies, sensitivity to class imbalance, high misclassification rate of single-stage models, and insufficient auxiliary explanatory information.

[0009] This invention differs from existing single-stage image classification methods. First, it performs standardized preprocessing and longitudinal registration on multimodal, multi-time-point images of the target object, constructing target regions, abnormal regions, historical abnormal regions, locally changing regions, surrounding extended regions, and longitudinal difference maps. Second, it constructs a basic image representation learning module and a longitudinal feature fusion module to extract spatial image features at each time point and dynamic change features of the same target object across different time points. Then, it employs a progressive multi-stage learning strategy, using sample scheduling from easy to difficult, classification tasks progressing from coarse to fine, input progression from single time point to multiple time points, class imbalance correction, and relearning of difficult samples to gradually obtain a stable risk representation. Finally, it uses the first-stage coarse-grained risk classifier for preliminary screening, inputting non-low-risk, suspected abnormal, abnormal extension related, or uncertain samples into the second-stage fine-grained risk classifier for further classification. Difficult samples are re-discriminated based on uncertainty scores, and the multi-stage outputs are fused to obtain the final risk classification information.

[0010] The method described in this invention mainly includes the following four steps: construction of multimodal and multitemporal image data, construction of a progressive multi-stage risk classification model, progressive training and model optimization, and multitemporal image risk classification and auxiliary interpretation output.

[0011] (1) Construction of multimodal and multitemporal image data

[0012] Images of the target object at at least one detection time point are acquired, preferably including a reference time point image, an earlier detection time point image, and at least one subsequent detection time point image. The images are then subjected to denoising, distortion correction, spatial resampling or scale unification, intensity normalization, multimodal registration, and longitudinal registration at different time points to align images of the same target object at different time points to a unified reference space. Furthermore, target regions, anomalous regions, historical anomalous regions, locally changed regions, severely anomalous regions, and surrounding extended regions are extracted, and local high-resolution image patches, overall context images, region masks, region statistical features, and longitudinal difference maps are generated.

[0013] For the image of the same target object at time point t and its reference time point image The longitudinal difference plot can be represented as:

[0014] (1)

[0015] Where Norm(·) represents the intensity normalization operation, (·) represents the spatial transformation of registering the image at time point t to the space of the reference time point. In practice, multi-channel longitudinal difference features can also be constructed based on the volume change rate of significant abnormal regions, the volume change rate of extended abnormal regions, the maximum diameter change rate of abnormal regions, and local intensity changes.

[0016] (2) Construction of a progressive multi-stage risk classification model

[0017] A basic image representation learning module is constructed to extract spatial image features from images at each time point; a longitudinal feature fusion module is constructed to fuse spatial image features and time interval codes from different time points; an auxiliary variable encoding module is constructed to encode auxiliary attribute information such as object attributes, operating parameters, and maintenance records into auxiliary variable feature vectors; a first-stage coarse-grained risk classifier and a second-stage fine-grained risk classifier are constructed to complete coarse screening and fine classification tasks, respectively.

[0018] The basic image representation learning module can employ a three-dimensional convolutional neural network, a two-dimensional convolutional neural network, a visual Transformer, a Mamba state space model, or a convolutional attention hybrid network; the vertical feature fusion module can employ a gated recurrent unit, a long short-term memory network, a Transformer encoder, a Mamba state space module, a temporal attention module, or a one-dimensional temporal convolution module.

[0019] The first-stage coarse-grained risk classifier outputs coarse-grained classification probabilities of low risk versus non-low risk, stable versus suspected anomaly, or no expansion versus expansion. The second-stage fine-grained risk classifier receives high-resolution local anomaly image patches, surrounding expansion region images, longitudinal difference maps, and auxiliary variable features, and outputs probabilities of medium risk, high risk, transient anomaly, true anomaly expansion, preset failure risk level, or other preset fine-grained risk categories.

[0020] (3) Progressive training and model optimization

[0021] To mitigate the impact of class imbalance and difficult samples on model training stability, this invention calculates a sample difficulty score for each training sample and dynamically adjusts the sample admission threshold based on training epochs, enabling the model to progressively learn from typical samples to boundary samples, minority class samples, and uncertain samples. The sample difficulty score D_i can be calculated using the following formula:

[0022] (2)

[0023] in, This represents the current training loss for the i-th sample. Represents the prediction probability entropy. This indicates consistency in expert annotation. Indicates the intensity of vertical variation in the target anomaly region. Indicates the scarcity of a category. to The weight coefficients are non-negative and Predictive probability entropy It can be calculated using the following formula:

[0024] (3)

[0025] Where K is the number of categories, Let be the predicted probability that the i-th sample belongs to the k-th class. During the e-th training round, the sample admission threshold is... Update using the following formula:

[0026] (4)

[0027] Where E is the total number of training rounds, The threshold growth coefficient is preferably between 1 and 3. The preferred value is 0.20 to 0.40. The optimal value is between 0.80 and 1.00. Only select [the appropriate value] during the initial training phase. Not higher than The samples were used for training, and then the range of training samples was gradually expanded.

[0028] To alleviate class imbalance, this invention sets dynamic class weights based on the number of samples in each class. The weight of class k in the e-th training round... It can be calculated using the following formula:

[0029] (5)

[0030] Where N is the total number of training samples, and K is the number of classes. Let be the number of samples in the k-th class. In the early stages of training, the minority class has a higher weight, and in the later stages of training, it gradually approaches the distribution of real application scenarios, thereby achieving a balance between minority class recognition ability and overall probability calibration.

[0031] The model training uses a joint loss function, which takes the following form:

[0032] (6)

[0033] in, For coarse-grained classification loss, For fine-grained classification loss, For vertical consistency loss, Loss due to uncertainty constraints For probability calibration loss, to The weights are set to loss weights. Preferably, the number of training epochs is set to 80 to 200, the initial learning rate is set to 1e-5 to 1e-3, the batch size is set to 1 to 8, the optimizer is AdamW or SGD, and the weight decay is set to 1e-6 to 1e-4.

[0034] (4) Risk classification and auxiliary interpretation output of multi-temporal images

[0035] During the inference phase, the preprocessed image, region mask, longitudinal difference map, and auxiliary variables are first input into the basic image representation learning module, the longitudinal feature fusion module, and the first-stage coarse-grained risk classifier. If the first-stage output is low risk and the confidence level is higher than a preset threshold, a low-risk auxiliary classification result is output; if the first-stage output is not low risk, suspected anomaly, anomaly extension correlation, or uncertain, the sample is input into the second-stage fine-grained risk classifier.

[0036] To identify low-confidence samples and samples with inconsistencies between stages, this invention calculates the model uncertainty score. Its form is as follows:

[0037] (7)

[0038] in, To predict probability entropy, The probability variance obtained from model ensemble or Monte Carlo Dropout, This is an indicator of inter-stage inconsistency; it is set to 1 when the classification results of the first stage and the second stage are inconsistent, and 0 otherwise. The highest category probability, to The weight coefficients are non-negative and .like Greater than the uncertainty threshold or Below the confidence threshold When this occurs, the difficult sample re-discrimination module is triggered. Preferably, Take a value between 0.45 and 0.70. Take a value between 0.50 and 0.80.

[0039] Final risk probability It can be obtained by weighted fusion of the first-stage mapping probability, the second-stage fine-classification probability, the hard sample re-discrimination probability, and the longitudinal risk probability:

[0040] (8)

[0041] Among them, to To merge weights, and The final output includes risk categories, probabilities for each category, model confidence scores, uncertainty scores, heatmaps of key image regions, and longitudinal change cues. Key image regions can be generated using attention weights, gradient-weighted activation maps, saliency maps, or region contribution scores; longitudinal change cues include one or more of the following: changes in significant anomalous regions, changes in expanding anomalous regions, changes in anomalous region volume, changes in surrounding expanding regions, and changes in anomalous region boundaries.

[0042] Compared with existing technologies, this invention has the following advantages and beneficial effects: First, by constructing a difference map through longitudinal registration, this invention explicitly utilizes image change information from multiple time points, which is beneficial for reflecting the dynamic changes in the state of the same target object over time. Second, this invention adopts a multi-stage classification mechanism from coarse to fine, directly outputting high-confidence, low-risk samples, and performing fine-grained classification and re-discrimination on non-low-risk or uncertain samples, reducing the risk of misjudgment of boundary samples. Third, this invention achieves progressive training from easy to difficult and from balanced to true distribution through sample difficulty scoring, dynamic sample admission thresholds, and class weight adjustments, improving the model training stability and minority class recognition ability. Fourth, this invention triggers re-discrimination of difficult samples through uncertainty scoring and inter-stage consistency judgment, enabling focused review of low-confidence samples. Fifth, this invention not only outputs risk categories but also outputs probabilities, confidence levels, key image regions, and longitudinal change prompts, facilitating manual review and auxiliary judgment. Attached Figure Description

[0043] Figure 1 Flowchart of a method for constructing a multi-temporal image risk classification model

[0044] Figure 2 Schematic diagram of progressive multi-stage learning model structure and training strategy

[0045] Figure 3 Uncertainty-driven risk re-judgment and auxiliary explanation output diagram Detailed Implementation

[0046] Based on the above description, a specific implementation process of the present invention is given below, using multi-temporal image risk classification of industrial components as an example for illustration. However, the scope of protection of the present invention is not limited to this implementation process. Any equivalent substitutions, parameter adjustments, module replacements, or scene extensions made based on the technical concept of the present invention should fall within the scope of protection of the present invention.

[0047] Step 1: Construction of multimodal and multitemporal image data

[0048] Step 1.1: Acquire multi-temporal image data of the target object. In this embodiment, the target object is an industrial component, and each industrial component includes a baseline time point detection image, an early detection time point image, and at least one subsequent detection time point image. Each time point includes at least one of visible light images, infrared thermograms, X-ray images, ultrasonic images, depth maps, and 3D point cloud images; when conditions permit, it may further include industrial endoscopic images, eddy current imaging, or other non-contact detection images. Next, acquire auxiliary attribute information. In this embodiment, auxiliary attribute information includes component type, material type, service life, load level, temperature, humidity, vibration parameters, pressure parameters, rotational speed parameters, maintenance records, historical risk level, detection time interval, and subsequent status labels. The above auxiliary attribute information is processed for missing values ​​and numerically standardized before being input into the auxiliary variable encoding module.

[0049] Step 1.2: Perform image preprocessing. Convert images of different formats to a preset format, and perform denoising, distortion correction, scale unification, and intensity normalization on each image modality. Intensity normalization preferably uses z-score standardization within the effective region of the target object, i.e. ,in and These represent the mean and standard deviation of the intensity within the effective area of ​​the target object, respectively.

[0050] Step 1.3: Perform multimodal registration and longitudinal registration. Register different image modalities at the same time point to the visible light image space, 3D point cloud space, or a preset target object coordinate system; register images of the same target object at different time points to the reference time point image space, early detection time point image space, or a preset template space. After registration, generate comparable image data between time points.

[0051] Step 1.4: Extract risk-related regions. Use automatic segmentation networks, semi-automatic segmentation tools, or manual annotation to obtain masks for anomalous regions, historical anomalous regions, locally changed regions, severely anomalous regions, and surrounding extended regions. Surrounding extended regions can be obtained by extending the boundaries of the anomalous regions outwards by a certain scale. Crop two-dimensional image blocks, three-dimensional volume data blocks, or local point cloud blocks centered on the anomalous regions and surrounding extended regions, while preserving the downsampled overall context image.

[0052] Step 1.5: Construct longitudinal difference features. Perform difference calculations on images of the same target object at adjacent time points or reference time points to obtain the rate of change of abnormal area, the rate of change of defect length, the rate of change of abnormal volume, the local intensity change map, and the regional morphology change map.

[0053] Step 2: Construction of a progressive multi-stage risk classification model

[0054] Step 2.1: Construct the basic image representation learning module. In this embodiment, the basic image representation learning module uses a three-dimensional convolutional neural network as the backbone network, but it can also be replaced by a visual Transformer, a Mamba state-space model, or a convolutional attention hybrid network. Multimodal images, region masks, and local image patches are concatenated along the channel dimension and input into the network, outputting the spatial image features at each time point. .

[0055] Step 2.2: Construct a longitudinal feature fusion module. This involves fusing spatial image features from each time point. With time interval encoding After being stitched together, the image is input into a time-series encoder to obtain a longitudinal image representation. Time interval encoding Used to indicate the interval between detection time and a baseline or reference detection time point; the unit can be days, weeks, or months.

[0056] Step 2.3: Construct the auxiliary variable encoding module. Input auxiliary attribute information such as material properties, environmental parameters, load parameters, operating parameters, service life, and maintenance records into the multilayer perceptron to obtain auxiliary variable features. .Will and By splicing or attention fusion, a comprehensive risk characterization can be obtained. .

[0057] Step 2.4: Construct the first-stage coarse-grained risk classifier. The first-stage classifier receives the comprehensive risk representation. Output coarse-grained classification probabilities for low-risk and non-low-risk, stable and suspected abnormal, or no-expansion and expanded correlation. If the low-risk probability is greater than the first confidence threshold. If the probability of a non-low-risk outcome is greater than the second confidence threshold, then a preliminary low-risk result will be output directly; otherwise, the probability of a non-low-risk outcome will be output directly. If the uncertainty score exceeds a preset threshold, then the process proceeds to the second stage of fine-grained classification. Preferably, and Take a value between 0.60 and 0.85.

[0058] Step 2.5: Construct the second-stage fine-grained risk classifier. The second-stage classifier receives high-resolution anomaly region image patches, surrounding extended region images, longitudinal difference maps, regional statistical features, and auxiliary variable features, and outputs the probabilities of medium-risk, high-risk, transient anomaly, true anomaly extension, or preset risk categories. .

[0059] Step 2.6: Construct the difficult sample re-discrimination module. This module receives the second-stage fine-classification results, the first-stage coarse-classification results, the longitudinal difference plot, and auxiliary variable features. It recalculates the risk probability for low-confidence samples, inter-stage inconsistent samples, and boundary samples, and outputs the re-discrimination probability. .

[0060] Step 3: Progressive Training and Model Optimization

[0061] Step 3.1: Calculate the sample difficulty score. Calculate the sample difficulty score for each sample before training begins or after each training cycle. . The weighted average of training loss, prediction probability entropy, expert annotation consistency, longitudinal change intensity of the target anomaly region, and category scarcity is obtained, as shown in formula (2). Expert annotation consistency The consistency rate or consistency coefficient can be determined by multiple manual annotations; category scarcity. Normalization can be performed based on the reciprocal of the number of samples belonging to a category.

[0062] Step 3.2: Perform sample scheduling from easy to difficult. In the early stages of training, only select... Not higher than The training module is based on typical samples for training the basic image representation learning module and the first-stage coarse-grained risk classifier. In the middle of training, medium-difficulty samples and minority class samples are added, and the second-stage fine-grained risk classifier is started. In the later stage of training, all samples are added, and hard sample relearning and probability calibration are enabled. According to formula (4), it increases with the number of training rounds.

[0063] Step 3.3: Perform input progression from single time point to multiple time points. In the early stage of training, a base classification model is trained using images from a single detection time point; in the middle stage of training, baseline time point images and early detection time point images are introduced; in the later stage of training, one or more subsequent detection time point images are introduced, and the longitudinal feature fusion module is enabled so that the model can gradually learn the dynamic features of the target object's state changing over time.

[0064] Step 3.4: Perform class imbalance correction. Calculate the dynamic class weights according to formula (5). The training weights are increased for minority class samples, transient abnormal samples, extended samples of real abnormalities, or high-risk samples. A strong class balancing strategy is used in the early stages of training, and the model gradually reverts to the distribution of real application scenarios in the later stages of training to reduce the bias of the model's probability output.

[0065] Step 3.5: Perform joint optimization. Train the model using the joint loss function shown in Equation (6). The coarse-grained classification loss and fine-grained classification loss can be cross-entropy loss or focus loss; the longitudinal consistency loss is used to constrain the continuity of risk representation of the same target object at adjacent time points; the uncertainty constraint loss is used to improve the model's ability to identify low-confidence samples; the probability calibration loss can be temperature scaling or calibration error constraint.

[0066] Step 3.6: Update the set of difficult samples. After each training epoch, the set of difficult samples with incorrect predictions and the highest probability of the class being lower than [a certain value] will be updated. Predicted entropy is higher than Samples with inconsistent classification results between stages are added to the hard sample set, and their sampling probability or loss weight is increased in subsequent training. Preferably, Take a value between 0.50 and 0.80. Take a value between 0.40 and 0.70.

[0067] Step 4: Multi-temporal image risk classification and auxiliary interpretation output

[0068] Step 4.1: Input the target object data to be analyzed. Preprocess, register, extract regions, and construct longitudinal difference features for the multi-temporal image data and auxiliary attribute information of the target object according to Step 1.

[0069] Step 4.2: Perform the first-stage coarse-grained classification. Input the processed image and auxiliary variable features into the basic image representation learning module, the longitudinal feature fusion module, and the first-stage coarse-grained risk classifier to obtain the coarse-grained risk probability. .like If the high-confidence, low-risk output condition is met, then low-risk auxiliary information is output; otherwise, proceed to the second stage of fine-grained classification.

[0070] Step 4.3: Perform the second-stage fine-grained classification. For samples that are not low-risk, suspected anomalies, extendedly relevant, or uncertain, input them into the second-stage fine-grained risk classifier to obtain the fine-grained risk probability. .

[0071] Step 4.4: Perform uncertainty assessment. Calculate the model uncertainty score according to formula (7). .when Greater than the uncertainty threshold Or the highest category probability Below the confidence threshold If the sample is in the difficult sample re-discrimination module, then input the sample into the module; otherwise, according to... and Perform result fusion.

[0072] Step 4.5: Perform hard sample re-discrimination. The hard sample re-discrimination module rereads the high-resolution anomaly region image patch, the surrounding extended region image, the longitudinal difference map, and the auxiliary variable features, and outputs the re-discrimination probability. It is used to correct the original fine classification results.

[0073] Step 4.6: Perform multi-stage result fusion. Following formula (8), fuse the first-stage mapping probability, the second-stage fine-classification probability, the hard sample re-discrimination probability, and the longitudinal risk probability to obtain the final risk probability. The final risk category is determined by... The category corresponding to the highest probability is determined.

[0074] Step 4.7: Output auxiliary explanation information. Output the final risk category, the probability of each category, the model confidence score, the uncertainty score, the key image region heatmap, and the longitudinal change prompt information. Among them, the key image region heatmap is used to indicate the main abnormal regions, historical abnormal regions, local change regions, or surrounding expansion regions on which the model is based; the longitudinal change prompt information is used to indicate the changes in the area of ​​abnormal regions, the changes in local intensity, the changes in boundary expansion, and the changes in surrounding expansion regions.

[0075] Step 4.8: Generate supplementary explanatory text. Based on the risk category, category probability, model uncertainty score, key image regions, and longitudinal change prompts, generate explanatory text for manual review. For example, when the model outputs a high risk and the key regions are concentrated at the anomaly boundary, while the longitudinal difference map shows that the area of ​​the anomaly region and its surrounding expansion region are continuously increasing, the system generates corresponding risk warning text. This explanatory text is for supplementary reference only and does not replace the final human judgment.

Claims

1. A method for constructing and applying a multi-temporal image risk classification model, characterized in that: First, acquire multimodal and multitemporal image data and auxiliary attribute information of the target object. The multimodal and multitemporal image data includes images at least one detection time point, including a reference time point image, an early detection time point image, and at least one subsequent detection time point image. The images include one or more of the following: visible light images, infrared thermograms, X-ray images, ultrasound images, depth maps, three-dimensional point cloud images, or industrial endoscopic images. Secondly, the image is subjected to standardization preprocessing, multimodal registration and longitudinal registration to extract the target region, abnormal region, historical abnormal region, local change region, severe abnormal region and surrounding extended region, and longitudinal difference map, regional change rate and time interval encoding are constructed based on images at different time points; Subsequently, a progressive multi-stage risk classification model is constructed, which includes a basic image representation learning module, a longitudinal feature fusion module, an auxiliary variable encoding module, a first-stage coarse-grained risk classifier, a second-stage fine-grained risk classifier, an uncertainty assessment module, and a difficult sample re-discrimination module. Then, the risk classification model is optimized using a progressive training strategy, which includes sample scheduling from easy to difficult, input progression from single time point to multiple time points, task progression from coarse to fine, dynamic correction of class imbalance, and updating of difficult sample sets. Finally, in the inference stage, the first-stage coarse-grained risk classifier is used to perform preliminary risk screening on the samples to be analyzed. Samples that are not low-risk, suspected abnormal, abnormally related, or uncertain are input into the second-stage fine-grained risk classifier. When the uncertainty score exceeds the preset threshold, the difficult sample re-discrimination module is triggered. The outputs of the multiple stages are fused to obtain the final risk classification information. The final risk classification information includes risk category, probability of each category, model confidence level, uncertainty score, heat map of key image area and longitudinal change prompt information, and the final risk classification information is only used as auxiliary reference information for manual review.

2. The method according to claim 1, characterized in that: Step 1: Construction of multimodal and multitemporal image data Step 1.1: Acquire multi-temporal image data and auxiliary attribute information of the same target object, wherein the target object is an industrial component. The multi-temporal image data includes a reference time point image, a post-maintenance or early inspection time point image, and one or more subsequent inspection time point images. Each time point includes at least one of visible light image, infrared thermogram, X-ray image, ultrasonic image, depth map, and 3D point cloud image, and may further include industrial endoscopic image, eddy current testing image, or other non-destructive testing image. The auxiliary attribute information includes one or more of the following: object type, material properties, structural dimensions, service time, environmental parameters, load parameters, operating parameters, historical maintenance records, historical risk level, inspection time interval, and subsequent status label. Step 1.2: Denoise the image, correct distortion, unify resolution, normalize scale, and normalize intensity; The intensity normalization is achieved by standardizing the z-score within the effective region of the target object. ,in The average intensity within the effective area of ​​the target object. The standard deviation of the intensity within the effective area of ​​the target object; Step 1.3: Register different image modalities at the same time point to the visible light image space, infrared thermal image space, or 3D point cloud space; register images of the same target object at different time points to the early detection time point image space, reference time point image space, or preset object template space; use automatic segmentation networks, semi-automatic segmentation tools, or manual annotation methods to obtain masks for abnormal regions, historical abnormal regions, local change regions, severely abnormal regions, and peripheral expansion regions. The peripheral expansion region is the area formed by extending outward from the boundary of the abnormal region by 5mm to 20mm. Step 1.4: Crop local image blocks or 3D data blocks centered on the abnormal region and its surrounding extended region; and preserve the downsampled global context image, region mask, and region statistical features; Step 1.5: Construct longitudinal difference features based on the registered images at different time points. The longitudinal difference features include longitudinal difference maps. Regional change rate One or more of the following: local intensity variation map and abnormal boundary change rate; among which , Norm(·) represents the intensity normalization operation. (·) represents the spatial transformation of registering the image at time point t to the space of the reference time point. This represents the area, volume, or equivalent size of the target anomaly region at time point t. This indicates the area, volume, or equivalent size of the target anomaly region at a reference time point. It is the stability constant.

3. The method according to claim 1, characterized in that: Step 2: Construction of a progressive multi-stage risk classification model Step 2.1: Construct a basic image representation learning module. The multimodal image, region mask, local image patch, and global context image are concatenated along the channel dimension and input into the basic image representation learning module to extract spatial image features at each time point. The basic image representation learning module employs one or more of the following: a three-dimensional convolutional neural network, a two-dimensional convolutional neural network, a visual Transformer, a Mamba state-space model, or a convolutional attention hybrid network. Step 2.2: Construct a vertical feature fusion module to integrate spatial image features from each time point. With time interval encoding After being stitched together, the image is input into a time-series encoder to obtain a longitudinal image representation. The time series encoder employs one or more of the following: gated recurrent unit, long short-term memory network, Transformer encoder, Mamba state space module, temporal attention module, or one-dimensional temporal convolution module. Step 2.3: Construct an auxiliary variable encoding module. Input one or more auxiliary variables from the following categories—object type, material properties, structural dimensions, environmental parameters, load parameters, operating parameters, service time, and historical maintenance records—into the multilayer perceptron to obtain auxiliary variable features. and will and A comprehensive risk representation is generated through splicing, weighted summation, or attention fusion. ; Step 2.4: Construct the first-stage coarse-grained risk classifier, which receives... Output coarse-grained classification probability The coarse-grained classification includes one or more of the following: low risk and non-low risk, stable and suspected abnormal, non-extended and extended correlation, normal and abnormal; when the low-risk probability is greater than the first confidence threshold. Outputs preliminary low-risk results when the probability of non-low risk exceeds the second confidence threshold. If the uncertainty score exceeds a preset threshold, the system will proceed to the second stage of fine-grained classification. and Select values ​​between 0.60 and 0.85; Step 2.5: Construct the second-stage fine-grained risk classifier and hard sample re-discrimination module; the second-stage fine-grained risk classifier receives high-resolution local anomaly image patches, surrounding extended region images, longitudinal difference maps, regional statistical features, and auxiliary variable features, and outputs fine-grained classification probabilities for low risk, medium risk, high risk, transient anomaly, true anomaly extension, or preset failure risk level. The difficult sample re-discrimination module receives the first-stage coarse classification results, the second-stage fine classification results, the longitudinal difference map, and auxiliary variable features, and outputs the re-discrimination probability for low-confidence samples, inter-stage inconsistent samples, and boundary samples. .

4. The method according to claim 1, characterized in that: Step 3: Progressive Training and Model Optimization Step 3.1: Calculate the sample difficulty score for each sample in the training set. , ,in This represents the current training loss for the i-th sample. Represents the prediction probability entropy. This indicates consistency in expert annotation. Indicates the intensity of vertical variation in the target anomaly region. Indicates the scarcity of a category. to The weight coefficients are non-negative and ;in K is the number of categories. Let be the predicted probability that the i-th sample belongs to the k-th class; Step 3.2: Dynamically adjust the sample admission threshold according to the training rounds. , Where e is the current training round, and E is the total number of training rounds. The threshold growth coefficient is selected from 1 to 3. Select a value between 0.20 and 0.

40. Select from 0.80 to 1.00; when Not higher than When the i-th sample is included in the current round of training, the training samples are gradually expanded from typical samples to boundary samples, minority class samples and uncertain samples. Step 3.3: Perform input progression from single time point to multiple time points and task progression from coarse to fine; In the early stage of training, a single detection time point image is used to train the basic image representation learning module and the first-stage coarse-grained risk classifier. In the middle stage of training, a baseline time point image and early detection time point images are introduced to train the second-stage fine-grained risk classifier. In the later stage of training, one or more subsequent detection time point images are introduced and the vertical feature fusion module, hard sample relearning and probability calibration are enabled. Step 3.4: Set dynamic class weights based on the number of samples in each class. , Where N is the total number of training samples and K is the number of classes. Let k be the number of samples in the k-th class. In the early stages of training, minority class samples are given higher training weights, and in the later stages of training, the class weights are gradually made closer to the distribution of real application scenarios, so as to achieve a balance between minority class recognition ability and overall probability calibration. Step 3.5: Train the model using a joint loss function. ,in For coarse-grained classification loss, For fine-grained classification loss, For vertical consistency loss, Loss due to uncertainty constraints For probability calibration loss, to The training epochs were set to 80 to 200, the initial learning rate was set to 1e-5 to 1e-3, the batch size was set to 1 to 8, the optimizer was AdamW or SGD, and the weight decay was set to 1e-6 to 1e-4. Step 3.6: After each training cycle, identify classes with incorrect predictions and highest class probabilities below the confidence threshold. Predicted entropy is higher than the entropy threshold Samples with inconsistent classification results between stages are added to the hard sample set, and their sampling probability or loss weight is increased in subsequent training. Select a value between 0.50 and 0.

80. Select values ​​between 0.40 and 0.

70.

5. The method according to claim 1, characterized in that: Step 4: Risk Classification and Supporting Explanation Output Step 4.1: Preprocess, register, extract regions, and construct longitudinal differential features from the multi-temporal images and auxiliary attribute information of the target object to be analyzed according to Step 1, and input them into the basic image representation learning module, the longitudinal feature fusion module, and the first-stage coarse-grained risk classifier to obtain the coarse-grained risk probability. ;like If the high-confidence, low-risk output condition is met, then output low-risk auxiliary information; otherwise, proceed to the second stage of fine-grained classification. Step 4.2: For samples that are not low-risk, suspected anomalies, anomaly extension related to, or uncertain, input the high-resolution local anomaly image patch, the surrounding extended region image, the longitudinal difference map, regional statistical features, time interval encoding, and auxiliary variable features into the second-stage fine-grained risk classifier to obtain the fine-grained risk probability. ; Step 4.3: Calculate the model uncertainty score , ,in To predict probability entropy, The probability variance obtained from model ensemble or Monte Carlo Dropout, This is an indicator of inter-stage inconsistency; it is set to 1 when the classification results of the first stage and the second stage are inconsistent, and 0 otherwise. The highest category probability, to The weight coefficients are non-negative and ;when Greater than the uncertainty threshold or Below the confidence threshold At that time, the sample is input into the difficult sample re-discrimination module. Select values ​​between 0.45 and 0.

70. Select a value between 0.50 and 0.80; Step 4.4: The hard sample re-discrimination module re-reads the high-resolution local anomaly image patch, the surrounding extended region image, the longitudinal difference map, and the auxiliary variable features, and outputs the re-discrimination probability. Final risk probability Mapping probability from the first stage Second-stage sub-classification probability Difficult sample re-discrimination probability and vertical risk probability The weighted fusion is obtained, ,in to To integrate weights and ; Step 4.5: According to Output the final risk category, probability of each category, model confidence and uncertainty score, and generate key image region heatmaps based on attention weights, gradient-weighted class activation maps, saliency maps or regional contribution scores; at the same time, output one or more longitudinal change prompts from abnormal region intensity changes, region area changes, local deformation changes, changes in surrounding extended regions and abnormal boundary changes, and generate auxiliary explanatory text for manual review.

Images