Target identification and damage evaluation method and system based on photoelectric pod multi-mode perception

Abstract

The invention discloses a target identification and damage assessment method and system based on photoelectric pod multi-mode perception. The method comprises the steps that images before and after target damage are acquired, and a multi-source damage data set is constructed and preprocessed; carrying out multi-task processing on the preprocessed image through a deep learning detection network, and synchronously realizing target detection and preliminary assessment of a damage level; cross-frame correlation matching is carried out on the targets before and after damage, and comparison areas of the same target before and after damage are extracted; extracting multi-dimensional damage features of the contrast area, and constructing multi-dimensional feature vectors of the target before and after damage so as to comprehensively characterize the damage state of the target; and inputting the preliminary assessment result and the multi-dimensional feature vector into a cascade classification decision model, and outputting a final damage assessment result. The method can be widely applied to scenes such as photoelectric pod reconnaissance, industrial facility fault detection and natural disaster damage analysis, and full-process automatic processing from target detection to damage evaluation is achieved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent assessment technology, and particularly relates to the field of accurate target recognition and intelligent damage level determination technology of multi-scene image and video data, especially a target recognition and damage assessment method and system based on multimodal perception of photoelectric pod. Background Technology

[0002] Traditional assessment methods mainly suffer from two major technical bottlenecks: First, they rely on manual on-site surveys or the deployment of precision sensors to measure mechanical parameters such as deflection-to-span ratio and rotation angle, which is difficult and inefficient to implement in hazardous environments. Second, existing intelligent assessment methods based on pre-trained models generally suffer from insufficient generalization ability across target types, and the assessment accuracy drops significantly when applied to untrained building or structural types.

[0003] Existing technologies mainly attempt to solve the above problems through two paths:

[0004] Physical mechanism driven path: CN120355776A patent ("A rapid damage assessment method for typical engineering targets based on image recognition technology") achieves non-contact assessment by establishing a mapping relationship between image damage features and damage level. This method requires obtaining damage images and blast resistance data, classifying damage levels and drawing PI curves; using cyclic interpolation to solve the critical damage conditions and fitting the critical damage curve expression; obtaining the critical damage area through simulation and establishing a damage assessment criterion based on image damage features with the damage area as the core. Its technical defects are: (1) It requires separate modeling for each type of engineering target, which is cumbersome and computationally expensive; (2) Image feature extraction is easily affected by the shooting environment and lacks stability in complex actual combat scenarios; (3) It is essentially a static criterion and cannot be adaptively updated using online data, resulting in poor adaptability to new structures or non-standard damage modes.

[0005] Data-driven approach: Patent CN119884848A ("Target damage assessment method and system, electronic device and storage medium") trains the model through building type adversarial loss function: the initial model is trained using a multi-building type sample set, and the damage classification loss and building type adversarial loss are used to enable the model to extract damage features that are not related to building type; then the classifier is fine-tuned using small sample data of target buildings. Its technical defects are: (1) it relies on a large amount of building sensor data, which cannot be applied to targets where no sensors are deployed or where data is missing; (2) model fine-tuning requires a certain amount of target domain labeled samples, which is difficult to obtain in emergency scenarios.

[0006] The two approaches mentioned above represent the "physical mechanism driven" and "data driven" technical routes, respectively, but neither has solved the core problem of achieving highly robust and rapidly deployable damage assessment under conditions of limited samples and unknown types. Summary of the Invention

[0007] The purpose of this invention is to address the problems existing in the prior art by proposing a target recognition and damage assessment method and system based on multimodal perception of an optoelectronic pod. This invention effectively overcomes the limitations of existing technologies through multimodal perception fusion and a dual-branch network design: it abandons reliance on physical parameter measurements and utilizes visible-infrared multimodal data to construct an adaptive assessment model; it achieves cross-type generalization without additional labeled samples through training on "pre-damage" and "post-damage" paired datasets; and it improves the accuracy of damage level assessment by 10%, while maintaining high precision on untrained target types.

[0008] The technical solution to achieve the objective of this invention is as follows: On the one hand, a target identification and damage assessment method based on multimodal perception of an optoelectronic pod is provided, the method comprising the following steps:

[0009] Step 1: Obtain images of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing;

[0010] Step 2: The preprocessed image is processed by a deep learning detection network to simultaneously achieve target detection and preliminary assessment of damage level.

[0011] Step 3: Perform cross-frame correlation matching on the target before and after damage, and extract the comparison area of ​​the same target before and after damage;

[0012] Step 4: Extract the multi-dimensional damage features of the comparison area and construct a multi-dimensional feature vector of the target before and after damage to comprehensively characterize the damage state of the target.

[0013] Step 5: Input the preliminary assessment results and multi-dimensional feature vectors into the cascaded classification decision model to output the final damage assessment results.

[0014] Furthermore, the deep learning detection network in step 2 adopts a dual-branch architecture, including:

[0015] The target detection branch is used to output the target category, location coordinates, and recognition confidence score.

[0016] Preliminary Damage Assessment Branch: Used to output the preliminary damage level and preliminary assessment confidence of the target based on the semantic features of the image;

[0017] The target detection branch and the preliminary damage assessment branch share the main feature extraction network, and a linkage mechanism for identification and assessment is constructed through feature reuse.

[0018] Furthermore, the backbone feature extraction network includes at least one of the following enhancement modules:

[0019] Hybrid Convolution Module: Multi-scale feature extraction and channel rearrangement fusion are performed through a dual-branch parallel structure of standard convolution and depthwise separable convolution;

[0020] Receptive field adaptive convolution module: Achieves adaptive feature enhancement in the spatial domain through a multi-branch receptive field structure.

[0021] Furthermore, the cross-frame matching in step 3 specifically includes:

[0022] Calculate the Intersection over Union (IOU) of targets before and after damage. If the IOU is greater than a preset threshold, they are determined to be the same target.

[0023] For targets with an IOU less than a preset threshold, a cost matrix combining category similarity and location distance is constructed using a state prediction algorithm, and the optimal matching association is solved using an assignment algorithm.

[0024] Furthermore, the multi-dimensional feature vector in step 4 includes at least the following two types of features:

[0025] Detection of change characteristics: including at least one or more of the following: target center displacement, aspect ratio change rate, and area change rate;

[0026] Image physical damage characteristics include at least one or more of the following: structural similarity, mean pixel difference, proportion of damaged area, contour change index, and edge preservation index.

[0027] Furthermore, the cascaded classification decision model described in step 5 includes:

[0028] The first-level classification unit is used to: process multi-dimensional feature vectors based on the random forest algorithm and output the initial damage level probability;

[0029] The second-level classification unit is used to achieve optimized classification by fusing the initial damage level probability with the original multi-dimensional feature vector based on the gradient boosting algorithm.

[0030] The decision-making unit is used to perform weighted voting on the preliminary evaluation results of the deep learning detection network and the output results of each level of classification unit to determine the final damage level.

[0031] Furthermore, the method also includes:

[0032] Step 6, Scene Adaptive Optimization; specifically including:

[0033] Stochastic scaling and data augmentation are introduced during the training phase of deep learning detection networks;

[0034] Temperature compensation is achieved through color space conversion and brightness / contrast adjustment.

[0035] On the other hand, a target recognition and damage assessment system is provided, the system including a data acquisition module, a multi-task processing module, a target matching module, a feature extraction module and a decision evaluation module;

[0036] The data acquisition module is used to acquire multimodal image data of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing.

[0037] The multi-task processing module is used to simultaneously perform target detection and preliminary damage assessment;

[0038] The target matching module is used to achieve accurate correlation of the target ROI before and after damage;

[0039] The feature extraction module is used to construct a multi-dimensional feature vector representing the damage state of the target;

[0040] The decision evaluation module is used to output comprehensive damage assessment results by applying a cascaded classification mechanism.

[0041] On the other hand, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the target recognition and damage assessment method based on optoelectronic pod multimodal perception when it invokes the computer program.

[0042] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the target recognition and damage assessment method based on optoelectronic pod multimodal perception.

[0043] Compared with the prior art, the significant advantages of this invention are:

[0044] (1) To address the problems of large computational redundancy and fragmented identification and assessment processes in existing methods, this invention designs a YOLOv11 dual-branch network to achieve simultaneous target identification and preliminary damage assessment. By reusing backbone network features to establish an "identification-assessment" linkage mechanism, the computational overhead is significantly reduced while ensuring accuracy, avoiding the independent computational processes of target identification and damage assessment in traditional methods.

[0045] (2) To address the limited representational capabilities of single features, this invention innovatively integrates target detection features and image physical damage features to construct a multi-dimensional feature vector that comprehensively represents the target damage state. The complementary enhancement of deep features and physical features improves the accuracy of damage level assessment by 10%, while maintaining a high accuracy of 85% on untrained target types, effectively overcoming the bottleneck of insufficient cross-type generalization ability.

[0046] (3) To address the problem of target association in occluded and multi-target scenarios, this invention proposes a target filtering and cross-frame matching strategy. By using the temporal association mechanism of "before-after damage" paired data, the continuity of damage feature extraction is ensured, which solves the target tracking problem in complex scenarios and avoids feature fragmentation caused by target drift in traditional methods.

[0047] (4) In order to improve the reliability of the evaluation results, the present invention designs a "cascaded classifier + weighted voting" decision mechanism, which performs multi-stage verification and integrated decision-making on the preliminary evaluation results and the secondary optimization results, significantly improving the evaluation confidence to above 0.85 and enhancing the system's fault tolerance under noise interference.

[0048] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0049] Figure 1 This is an overall flowchart of a target recognition and damage assessment method based on multimodal perception of an optoelectronic pod in one embodiment.

[0050] Figure 2 This is a system architecture diagram of the present invention in one embodiment.

[0051] Figure 3 This is a flowchart of a model training method in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0054] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0055] In one embodiment, a target identification and damage assessment method based on multimodal perception of an optoelectronic pod is provided, the method comprising the following steps:

[0056] Step 1: Based on typical application scenarios of optoelectronic pods, acquire images of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing.

[0057] Step 2: The preprocessed image is processed by a deep learning detection network to simultaneously achieve target detection and preliminary assessment of damage level.

[0058] Here, the damage level can be pre-defined according to the actual scenario or target;

[0059] Step 3: Perform cross-frame correlation matching on the target before and after damage, and extract the comparison area of ​​the same target before and after damage;

[0060] Step 4: Extract the multi-dimensional damage features of the comparison area and construct a multi-dimensional feature vector of the target before and after damage to comprehensively characterize the damage state of the target.

[0061] Step 5: Input the preliminary assessment results and multi-dimensional feature vectors into the cascaded classification decision model to output the final damage assessment results.

[0062] Furthermore, in one embodiment, the preprocessing in step 1 includes at least:

[0063] Image registration and normalization are performed on images of different modalities;

[0064] A timestamp synchronization mechanism is used to ensure that the time sequence interval of image frames before and after damage is within a preset threshold range.

[0065] Furthermore, in one embodiment, the deep learning detection network in step 2 adopts a dual-branch architecture, including:

[0066] The target detection branch is used to output the target category, location coordinates, and recognition confidence score.

[0067] Preliminary Damage Assessment Branch: Used to output the preliminary damage level and preliminary assessment confidence of the target based on the semantic features of the image;

[0068] The target detection branch and the preliminary damage assessment branch share the main feature extraction network, and a linkage mechanism for identification and assessment is constructed through feature reuse.

[0069] Here, the deep learning detection network adopts, but is not limited to, the YOLOv11 dual-branch network.

[0070] Preferably, in some embodiments, the backbone feature extraction network includes at least one of the following enhancement modules:

[0071] Hybrid Convolution Module: Multi-scale feature extraction and channel rearrangement fusion are performed through a dual-branch parallel structure of standard convolution and depthwise separable convolution;

[0072] Receptive field adaptive convolution module: Achieves adaptive feature enhancement in the spatial domain through a multi-branch receptive field structure.

[0073] Furthermore, in one embodiment, the cross-frame matching in step 3 specifically includes:

[0074] Calculate the Intersection over Union (IOU) of targets before and after damage. If the IOU is greater than a preset threshold, they are determined to be the same target.

[0075] For targets with an IOU less than a preset threshold, a cost matrix combining category similarity and location distance is constructed using a state prediction algorithm, and the optimal matching association is solved using an assignment algorithm.

[0076] Here, we are not limited to using the Intersection over Union (IOU); other metrics used to measure spatial overlap are also acceptable.

[0077] Furthermore, in one embodiment, the multi-dimensional feature vector in step 4 includes at least the following two types of features:

[0078] Detection of change characteristics: including at least one or more of the following: target center displacement, aspect ratio change rate, and area change rate;

[0079] Image physical damage characteristics include at least one or more of the following: structural similarity, mean pixel difference, proportion of damaged area, contour change index, and edge preservation index.

[0080] Furthermore, in one embodiment, the cascaded classification decision model in step 5 includes:

[0081] The first-level classification unit is used to process multi-dimensional feature vectors based on the random forest algorithm (such as the random forest classifier) ​​and output the initial damage level probability.

[0082] The second-level classification unit is used to perform optimized classification by fusing the initial damage level probability with the original multi-dimensional feature vector based on a gradient boosting algorithm (such as the LightGBM classifier).

[0083] The decision-making unit is used to perform weighted voting on the preliminary evaluation results of the deep learning detection network and the output results of each level of classification unit to determine the final damage level.

[0084] Furthermore, in one embodiment, the method further includes:

[0085] Step 6: Output the results, which should include at least the target category, damage assessment results, and overall confidence level, providing quantitative decision support for military strike and disaster emergency assessment.

[0086] The overall confidence level includes at least the confidence levels corresponding to deep learning detection networks, random forest algorithms, and gradient boosting algorithms.

[0087] Furthermore, in one embodiment, the method further includes:

[0088] Step 7, Scene Adaptive Optimization; specifically includes:

[0089] In the training phase of deep learning detection networks, random scaling and data augmentation are introduced to improve the robustness of the model against small targets and targets in harsh environments.

[0090] Temperature compensation is performed through color space conversion and brightness / contrast adjustment to eliminate image deviations in extreme environments.

[0091] In one embodiment, a target recognition and damage assessment system is provided, the system including a data acquisition module, a multi-task processing module, a target matching module, a feature extraction module, and a decision evaluation module;

[0092] The data acquisition module is used to acquire multimodal image data of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing.

[0093] The multi-task processing module is used to simultaneously perform target detection and preliminary damage assessment;

[0094] The target matching module is used to achieve accurate correlation of the target ROI before and after damage;

[0095] The feature extraction module is used to construct a multi-dimensional feature vector representing the damage state of the target;

[0096] The decision evaluation module is used to output comprehensive damage assessment results by applying a cascaded classification mechanism.

[0097] Specific limitations regarding the target recognition and damage assessment system can be found in the above description of the limitations for the target recognition and damage assessment method based on electro-optical pod multimodal perception, and will not be repeated here. Each module in the aforementioned target recognition and damage assessment system based on electro-optical pod multimodal perception can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0098] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program when invoked:

[0099] Step 1: Obtain images of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing;

[0100] Step 2: The preprocessed image is processed by a deep learning detection network to simultaneously achieve target detection and preliminary assessment of damage level.

[0101] Step 3: Perform cross-frame correlation matching on the target before and after damage, and extract the comparison area of ​​the same target before and after damage;

[0102] Step 4: Extract the multi-dimensional damage features of the comparison area and construct a multi-dimensional feature vector of the target before and after damage to comprehensively characterize the damage state of the target.

[0103] Step 5: Input the preliminary assessment results and multi-dimensional feature vectors into the cascaded classification decision model to output the final damage assessment results.

[0104] Furthermore, in one embodiment, when the processor invokes the computer program, it also performs:

[0105] Step 6: Output the results, which should include at least the target category, damage assessment results, and overall confidence level.

[0106] Furthermore, in one embodiment, when the processor invokes the computer program, it also performs:

[0107] Step 7, scene adaptive optimization.

[0108] For specific limitations on each step, please refer to the limitations on the target recognition and damage assessment method based on multimodal perception of optoelectronic pods mentioned above, which will not be repeated here.

[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0110] Step 1: Obtain images of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing;

[0111] Step 2: The preprocessed image is processed by a deep learning detection network to simultaneously achieve target detection and preliminary assessment of damage level.

[0112] Step 3: Perform cross-frame correlation matching on the target before and after damage, and extract the comparison area of ​​the same target before and after damage;

[0113] Step 4: Extract the multi-dimensional damage features of the comparison area and construct a multi-dimensional feature vector of the target before and after damage to comprehensively characterize the damage state of the target.

[0114] Step 5: Input the preliminary assessment results and multi-dimensional feature vectors into the cascaded classification decision model to output the final damage assessment results.

[0115] Furthermore, in one embodiment, when the processor invokes the computer program, it also performs:

[0116] Step 6: Output the results, which should include at least the target category, damage assessment results, and overall confidence level.

[0117] Furthermore, in one embodiment, when the processor invokes the computer program, it also performs:

[0118] Step 7, scene adaptive optimization.

[0119] For specific limitations on each step, please refer to the limitations on the target recognition and damage assessment method based on multimodal perception of optoelectronic pods mentioned above, which will not be repeated here.

[0120] Compared with existing technologies, this invention achieves significant improvements in target recognition accuracy, damage assessment reliability, and scene adaptability, specifically including:

[0121] Improved target recognition and evaluation efficiency: YOLOv11 achieves simultaneous recognition and preliminary evaluation through dual branches. Compared with the independent recognition + independent evaluation scheme, the feature extraction efficiency is improved by 20%, and the overall processing latency is reduced from 200ms to 120ms, meeting the requirements of real-time processing.

[0122] Improved damage assessment accuracy: Multi-dimensional feature vectors comprehensively represent the damage status, and combined with weighted voting decision-making, the damage assessment accuracy is ≥85%.

[0123] Enhanced adaptability to complex scenes: Through cross-frame matching (Hungarian + IOU) and incremental training, the target matching accuracy in occluded scenes is ≥85%, and the misjudgment rate of damage assessment for small targets is reduced to 15%.

[0124] Improved generalization ability: The cascaded classifier (random forest + LightGBM) has strong noise resistance, and the damage assessment accuracy fluctuates by ≤5% in scenes with blurred images and changing lighting.

[0125] As a specific example, the invention will be further described in detail with reference to the accompanying drawings.

[0126] Combination Figures 1 to 3 This embodiment provides a target recognition and damage assessment method based on multimodal perception of an optoelectronic pod, the method comprising:

[0127] S1, Constructing a multi-source damage dataset and performing preprocessing; specifically including:

[0128] S1-1, Data Acquisition and Compliance Processing

[0129] Based on typical application scenarios of optoelectronic pods, paired data of "before damage" and "after damage" were collected from multiple public channels; the scope of authorization for all public data was confirmed through a "CC license verification tool" to remove copyrighted material;

[0130] Specifically, based on typical application scenarios of optoelectronic pods, paired "before-after" damage data was collected from multiple channels, including publicly available military damage footage, military science videos, open-source battlefield footage, and war strike footage (only frames of damaged targets without personnel identification or geographic coordinates were retained to ensure data compliance). All publicly available data underwent a "CC license verification tool" to confirm the scope of authorization (for non-commercial research only), and copyright-restricted materials were removed.

[0131] S1-2, Data Filtering and Matching

[0132] Filtering rules: Remove blurry frames, frames without a target, and overexposed / underexposed frames (e.g., average brightness <50 or >200), and retain valid frames;

[0133] By synchronizing with timestamps to ensure that the time difference between consecutive frames is ≤100ms, "1 frame before the attack (T0)" and "1-3 frames after the attack (T1-T3)" are paired samples to construct a paired dataset containing "frame before damage + frame after damage + annotation information".

[0134] S1-3, Labeling System Design

[0135] Target recognition annotation: bounding box (xmin, ymin, xmax, ymax), class label and recognition confidence; xmin and ymin are the minimum x and y coordinates of the bounding box, respectively, and xmax and ymax are the maximum x and y coordinates of the bounding box, respectively;

[0136] Damage levels are classified into three categories:

[0137] Zero-degree damage: The target surface has no visible damage and the structure is intact;

[0138] Minor damage: The target surface shows a small number of cracks or local deformation, but does not affect the overall load-bearing capacity of the structure;

[0139] Severe damage: The target part of the structure is significantly damaged, loses its load-bearing capacity, and is difficult to repair;

[0140] S1-4, ROI Extraction

[0141] Based on labeled bounding boxes, target ROI regions (such as armored vehicle ROIs) are cropped from previous and next frames to reduce background noise interference and improve model training efficiency.

[0142] Here, the present invention uses a pairwise dataset of "before-after damage" and, unlike existing technologies (such as CN120355776A, etc.) that rely on static criteria, it achieves adaptive evaluation through time-series associated data, without the need for separate modeling for each type of target, thus solving the problem of insufficient cross-type generalization ability.

[0143] S2 employs a YOLOv11 dual-branch network to simultaneously perform target detection and preliminary damage assessment; specifically including:

[0144] S2-1, Constructing a YOLOv11 dual-branch network architecture

[0145] A CSPDarknet-based backbone network is adopted, embedding GSConv and RFAConv modules to construct a dual-branch output structure. The backbone network uses the CSPDarknet architecture to ensure the efficiency and completeness of feature extraction. Among them, the GSConv module realizes multi-scale feature extraction and efficient feature fusion; the RFAConv module realizes adaptive feature enhancement in the spatial domain through a multi-branch receptive field structure.

[0146] Here, the GSConv module specifically includes:

[0147] A dual-branch architecture is adopted to achieve multi-scale feature extraction and efficient feature fusion, specifically including:

[0148] Standard convolution branch:

[0149] F_std = Conv2d(X, K_std)

[0150] In the formula, X is the input feature map, K_std is the standard convolution kernel, Conv2d is the standard convolution, and F_std is the output of the standard convolution branch;

[0151] Depthwise separable convolution branches:

[0152] F_dw = DWConv(X, K_dw) → PWConv(F_dw, K_pw)

[0153] In the formula, DWConv is depthwise separable convolution, K_dw is the depthwise convolution kernel, PWConv is pointwise convolution, K_pw is the pointwise convolution kernel, and F_dw is the output of the depthwise separable convolution branch.

[0154] Feature fusion unit:

[0155] F_fused = Concat(F_std, F_dw)

[0156] In the formula, Concat is the feature concatenation operation, and F_fused is the output of feature fusion;

[0157] Channel rearrangement unit:

[0158] F_out = ChannelShuffle(F_fused)

[0159] In the formula, ChannelShuffle is the channel rearrangement operation, and F_out is the output of the channel rearrangement unit.

[0160] The GSConv module has the following advantages in damage assessment:

[0161] Multi-scale damage coverage: a comprehensive characterization from overall structural deformation to local surface damage

[0162] Feature diversity preservation: Balancing the preservation of information between global context and local details

[0163] Computational efficiency optimization: Reduce computational complexity through depthwise separable convolution to meet real-time requirements.

[0164] Here, the RFAConv module includes:

[0165] Multi-receptive-field feature extraction unit:

[0166] F_multi = [Conv3x3(X), Conv5x5(X)]

[0167] In the formula, Conv3x3 is a 3×3 convolution, Conv5x5 is a 5×5 convolution; F_multi is the output of the multi-receptive-field feature extraction unit;

[0168] Feature fusion unit:

[0169] F_fused = Concat(F_multi)

[0170] In the formula, Concat represents the feature concatenation operation, and F_fused is the output of the feature fusion unit;

[0171] Attention weight generation unit:

[0172] A = σ(FC(GAP(F_fused)))

[0173] In the formula, FC is a fully connected layer, GAP is global average pooling, σ is the sigmoid activation function, and A is the output of the attention weight generation unit;

[0174] Feature recalibration unit:

[0175] F_out = A ⊙ F_fused

[0176] In the formula, ⊙ represents point-by-point multiplication, and F_out is the output of the feature recalibration unit.

[0177] Here are the advantages of RFAConv in damage assessment:

[0178] Focus on key areas: Give higher weight to critical areas such as structural connections and load-bearing components.

[0179] Background suppression: Effectively reduces interference from complex backgrounds and improves the feature signal-to-noise ratio.

[0180] Scale adaptation: Simultaneously considers local minor damage and overall structural deformation.

[0181] S2-2, Design of dual-branch output

[0182] Object detection branch: Outputs object category, bounding box coordinates (x, y, w, h, where x / y is the center coordinate of the box, and w / h is the width and height of the box), and recognition confidence score;

[0183] Preliminary Damage Assessment Branch: Outputs preliminary damage level results (three categories: zero-degree damage, minor damage, and severe damage) and preliminary confidence level for the damaged target;

[0184] The two branches share the feature maps extracted by the backbone network, reduce computational redundancy through feature reuse, and establish a "recognition-evaluation" linkage mechanism;

[0185] S2-3, Train the YOLOv11 dual-branch network;

[0186] Transfer learning: Initializing the model with YOLOv11 weights based on the training data;

[0187] Mission fine-tuning: Focus on fine-tuning the damage assessment branch for 5 types of targets;

[0188] Construct a joint loss function for "detection loss + damage assessment loss":

[0189] L_total = L_detect + λ1·L_assess + λ2·L_regularization

[0190] in,

[0191] L_detect = α·L_cls + β·L_obj + γ·L_ciou

[0192] L_assess = -Σ[w_i·y_i·log(ŷ_i)]

[0193] In the formula, L_detect is the detection loss, L_assess is the damage assessment loss, and L_regularization is the regularization term; L_cls is the classification loss, L_obj is the target detection loss, L_ciou is the CIoU loss, α, β, and γ are the loss weight coefficients, which are 0.5, 0.5, and 1.0, respectively; w_i is the class weight, y_i is the true label, and ŷ_i is the predicted probability; λ1 and λ2 are the regularization coefficients.

[0194] The YOLOv11 dual-branch network is trained based on the joint loss function and the dataset S1.

[0195] Here, the present invention achieves simultaneous processing of target identification and damage assessment by sharing backbone network features (different from the problem of the separation of identification and assessment processes in the prior art). Feature reuse reduces computational overhead by 30%. At the same time, the collaborative design of GSConv and RFAConv ("breadth capture-depth focus" mechanism) enables the network to pay attention to macroscopic structural deformation and microscopic damage patterns at the same time, significantly improving feature quality.

[0196] S3, through target filtering and cross-frame matching, achieves accurate correlation between targets before and after damage; specifically including:

[0197] S3-1, Perform target matching

[0198] The Hungarian algorithm combined with an IOU threshold is used to associate targets between consecutive frames, specifically including:

[0199] (1) When the IOU of the target in the current frame is greater than or equal to the IOU threshold (e.g., greater than or equal to 0.7), it is determined to be the same target;

[0200] (2) Occlusion scene matching: If IOU < IOU threshold, calculate the combined cost of "category similarity + location distance" using the Hungarian algorithm;

[0201] First, calculate the cost matrix:

[0202] Cost_Matrix(i,j) = 1 - IOU(Kalman_prediction_i, detection_j)

[0203] In the formula, IOU is the intersection-union ratio, Kalman_prediction_i is the Kalman filter prediction box, detection_j is the detection box in the current frame, and Cost_Matrix(i,j) represents the comprehensive cost matrix;

[0204] State transition equations: x_{t|t-1} = F·x_{t-1} + w_t, P_{t|t-1} = F·P_{t-1}·F^T + Q

[0205] In the formula, x_{t|t-1} is the predicted state at time t, F is the state transition matrix, x_{t-1} is the state at time t-1, w_t is the process noise, P_{t|t-1} is the prediction covariance at time t, P_{t-1} is the covariance at time t-1, and Q is the process noise covariance.

[0206] Observation equation: z_t = H·x_t + v_t

[0207] In the formula, z_t is the observed value at time t, H is the observation matrix, x_t is the state at time t, and v_t is the observation noise.

[0208] Then, the following equation is solved using the Hungarian algorithm:

[0209] minΣCost_Matrix(i, π(i)), satisfying IOU≥ε_high

[0210] In the formula, π(i) is the matching function, and ε_high represents the IOU threshold;

[0211] S3-2, ROI Preprocessing

[0212] For the same target that is successfully matched, the region of interest "before-after damage" is extracted, and the following steps are performed in sequence: grayscale conversion (to eliminate color interference), 5×5 Gaussian filtering (to suppress image noise), and histogram equalization (to enhance the contrast of the damaged area), to provide high-quality data for subsequent multi-dimensional feature calculation.

[0213] Here, the present invention adopts the "Hungarian algorithm + IOU threshold" strategy to achieve target matching. Unlike existing technologies (such as the MAD algorithm used in CN117710321A), by combining the comprehensive cost of "category similarity + location distance", it effectively solves the target matching problem in occluded scenes and ensures the continuity of damage feature extraction.

[0214] S4, calculates a multi-dimensional feature vector to comprehensively characterize the target damage state; specifically including:

[0215] The multi-dimensional feature vector Feature_Vector is constructed as follows:

[0216] Feature_Vector = [Δx, Δy, Δ(w / h), ΔS / S0, SSIM, PD, DAR, CCI, ERI]

[0217] Where Δx, Δy, Δ(w / h), and ΔS / S0 are all detected change features, (Δx, Δy) is the target center position offset, Δ(w / h) is the target aspect ratio change, and ΔS / S0 is the target area change rate; SSIM, PD, DAR, CCI, and ERI are image physical features, where SSIM is structural similarity, PD is the mean pixel difference, DAR is the proportion of damaged area, CCI is the contour change index, and ERI is the edge preservation index.

[0218] The calculation formulas for each variable in the multi-dimensional feature vector (Feature_Vector) are as follows:

[0219] Δx = x_after - x_before, Δy = y_after - y_before

[0220] In the formula, x_before and y_before are the coordinates of the target center before damage, and x_after and y_after are the coordinates of the target center after damage;

[0221] Δ(w / h) = (w_after / h_after) - (w_before / h_before)

[0222] In the formula, w_before and h_before are the width and height of the target before damage, and w_after and h_after are the width and height of the target after damage;

[0223] ΔS / S0 = (S_after - S_before) / S_before

[0224] In the formula, S_before is the target area before damage, and S_after is the target area after damage;

[0225] SSIM = [l(x,y)]^α·[c(x,y)]^β·[s(x,y)]^γ

[0226] In the formula, l(x,y) = (2μ_x*μ_y + C1) / (μ_x² + μ_y² + C1), where μ_x and μ_y are the means of the images before and after damage, respectively, and C1 is a constant to prevent the denominator from being zero; c(x,y) = (2σ_x*σ_y + C2) / (σ_x² +σ_y² + C2), where σ_x and σ_y are the standard deviations of the images before and after damage, respectively, and C2 is a constant; s(x,y) = (σ_xy +C3) / (σ_x*σ_y + C3), where σ_xy is the covariance of the images before and after damage, and C3 is a constant; α, β, and γ are weighting coefficients, which are 0.01, 0.01, and 0.01, respectively.

[0227] PD = (1 / N)·Σ|I_after(i,j) - I_before(i,j)|

[0228] In the formula, N is the total number of pixels in the ROI region, I_after(i,j) is the pixel value of the image at position (i,j) after damage, and I_before(i,j) is the pixel value of the image at position (i,j) before damage.

[0229] DAR = (N_damage / N_total)·100%

[0230] In the formula, N_damage is the number of pixels in the damaged area, and N_total is the total number of pixels in the ROI area; where the damaged area Damage_Mask is segmented by Otsu threshold: T = Otsu(ΔI), Damage_Mask = ΔI > T, ΔI is the difference between the images before and after the damage, T is the Otsu threshold, and Damage_Mask is the mask of the damaged area;

[0231] CCI = 1 - (N_matched / min(N_before, N_after))

[0232] In the formula, N_before and N_after are the number of contour matching points before and after the damage, respectively, and N_matched is the number of matching points;

[0233] ERI = (L_after / L_before)·100%

[0234] In the formula, L_before and L_after are the target edge lengths before and after damage, respectively.

[0235] Here, the present invention constructs a feature vector that comprehensively represents the damage state of the target through multi-dimensional feature calculation (detection change features + image physical features). Compared with existing technologies (such as CN120355776A which only uses area features), it integrates multi-dimensional damage information such as structure, pixels, and contours, which significantly improves the evaluation accuracy.

[0236] S5 applies a cascaded classification decision-making mechanism to optimize damage level assessment results; specifically including:

[0237] S5-1, based on multi-dimensional feature vectors, performs first-level optimization through a random forest classifier and outputs the corresponding damage level assessment results;

[0238] S5-2, based on the damage level probability output by the multi-dimensional feature vector and the random forest classifier, performs a second-level optimization through the LightGBM classifier to output the corresponding damage level assessment result;

[0239] S5-3, weighted voting decision-making:

[0240] Final_Class = argmax(Final_Score)

[0241] in,

[0242] Final_Score = w1·YOLO_output + w2·RF_prob + w3·LGB_prob

[0243] In the formula, YOLO_output is the damage level assessment result output by the YOLOv11 dual-branch network, RF_prob is the damage level assessment result output by the random forest, LGB_prob is the damage level assessment result output by the LightGBM, w1, w2, and w3 are the weights of the corresponding terms, Final_Class represents the final optimized damage level assessment result, and argmax is the index of the maximum value.

[0244] Preferably, the random forest classifier parameters are set as follows: n_estimators=200, max_depth=20, min_samples_split=5, min_samples_leaf=2; the LightGBM classifier parameters are set as follows: num_leaves=30, learning_rate=0.05, feature_fraction=0.8; and the weighted voting decision weights are allocated as follows: w1=0.4 (preliminary YOLOv11 results), w2=0.3 (random forest results), and w3=0.3 (LightGBM results).

[0245] Here, the present invention adopts a cascaded decision chain of "YOLOv11 preliminary results → random forest → LightGBM" and integrates multi-level results through a weighted voting mechanism. Unlike existing technologies (such as the simple averaging of results in CN119884848A), this invention significantly improves the reliability of decision-making and increases the accuracy of damage level assessment by 10%.

[0246] S6 outputs target category, damage level (three categories: zero damage, minor damage, and severe damage) and overall confidence level, providing quantitative decision support for military strike and disaster emergency assessment;

[0247] Here, the specific formula for calculating the overall confidence level is:

[0248] Overall confidence level = weighted average calculation

[0249] That is: Confidence = (w1·YOLO_conf + w2·RF_conf + w3·LGB_conf) / (w1 +w2 + w3);

[0250] In the formula, YOLO_conf is the confidence score of YOLOv11, RF_conf is the confidence score of Random Forest, and LGB_conf is the confidence score of LightGBM.

[0251] S7 performs scene adaptive optimization, specifically including:

[0252] (1) Multi-scale training: Introduce random scale transformation (such as (0.5-1.5 times random scaling) during the training phase of YOLOv11 dual-branch network to enhance the model's ability to detect small targets;

[0253] (2) Data augmentation: Expand the dataset in S1 by at least the following data augmentation methods: adding Gaussian noise (σ=0.01), simulating smoke effects (transparency 0.3-0.6), and random rotation (±15°) to improve the robustness of the model under harsh conditions;

[0254] (3) Temperature compensation: Eliminate color deviation in low temperature environment through image preprocessing algorithm, including at least: color space conversion (RGB→HSV), brightness adjustment, and contrast enhancement.

[0255] Here, the present invention effectively improves the stability of the system under harsh conditions such as low temperature, smoke, and long distance through multi-scale training, data augmentation and temperature compensation. Compared with the prior art, the misjudgment rate is reduced to less than 10%.

[0256] In summary, this invention effectively overcomes the limitations of existing technologies through multimodal perception fusion and a dual-branch network design: it abandons reliance on physical parameter measurements and utilizes visible-infrared multimodal data to construct an adaptive evaluation model; it achieves cross-type generalization without additional labeled samples through training on paired "pre-damage" and "post-damage" datasets; and it improves the accuracy of damage level assessment by 10%, while maintaining high precision even on untrained target types. This invention can be widely applied to scenarios such as optoelectronic pod reconnaissance, military strike effect assessment, industrial facility fault detection, and natural disaster damage analysis, realizing fully automated processing from target detection to damage assessment.

[0257] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A target recognition and damage assessment method based on multimodal perception of an optoelectronic pod, characterized in that, The method includes the following steps: Step 1: Obtain images of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing; Step 2: The preprocessed image is processed by a deep learning detection network to simultaneously achieve target detection and preliminary assessment of damage level. Step 3: Perform cross-frame correlation matching on the target before and after damage, and extract the comparison area of ​​the same target before and after damage; Step 4: Extract the multi-dimensional damage features of the comparison area and construct a multi-dimensional feature vector of the target before and after damage to comprehensively characterize the damage state of the target. Step 5: Input the preliminary assessment results and multi-dimensional feature vectors into the cascaded classification decision model to output the final damage assessment results.

2. The target recognition and damage assessment method based on multimodal perception of an optoelectronic pod as described in claim 1, characterized in that, In step 2, the deep learning detection network adopts a dual-branch architecture, including: The target detection branch is used to output the target category, location coordinates, and recognition confidence score. Preliminary Damage Assessment Branch: Used to output the preliminary damage level and preliminary assessment confidence of the target based on the semantic features of the image; The target detection branch and the preliminary damage assessment branch share the main feature extraction network, and a linkage mechanism for identification and assessment is constructed through feature reuse.

3. The target recognition and damage assessment method based on multimodal perception of an optoelectronic pod as described in claim 2, characterized in that, The backbone feature extraction network includes at least one of the following enhancement modules: Hybrid Convolution Module: Multi-scale feature extraction and channel rearrangement fusion are performed through a dual-branch parallel structure of standard convolution and depthwise separable convolution; Receptive field adaptive convolution module: Achieves adaptive feature enhancement in the spatial domain through a multi-branch receptive field structure.

4. The target recognition and damage assessment method based on multimodal perception of an optoelectronic pod as described in claim 1, characterized in that, The cross-frame matching mentioned in step 3 specifically includes: Calculate the Intersection over Union (IOU) of targets before and after damage. If the IOU is greater than a preset threshold, they are determined to be the same target. For targets with an IOU less than a preset threshold, a cost matrix combining category similarity and location distance is constructed using a state prediction algorithm, and the optimal matching association is solved using an assignment algorithm.

5. The target recognition and damage assessment method based on multimodal perception of an optoelectronic pod according to claim 1, characterized in that, In step 4, the multi-dimensional feature vector must include at least the following two types of features: Detection of change characteristics: including at least one or more of the following: target center displacement, aspect ratio change rate, and area change rate; Image physical damage characteristics include at least one or more of the following: structural similarity, mean pixel difference, proportion of damaged area, contour change index, and edge preservation index.

6. The target recognition and damage assessment method based on multimodal perception of an optoelectronic pod according to claim 1, characterized in that, The cascaded classification decision model described in step 5 includes: The first-level classification unit is used to: process multi-dimensional feature vectors based on the random forest algorithm and output the initial damage level probability; The second-level classification unit is used to achieve optimized classification by fusing the initial damage level probability with the original multi-dimensional feature vector based on the gradient boosting algorithm. The decision-making unit is used to perform weighted voting on the preliminary evaluation results of the deep learning detection network and the output results of each level of classification unit to determine the final damage level.

7. The target recognition and damage assessment method based on multimodal perception of an optoelectronic pod according to claim 1, characterized in that, The method further includes: Step 6, Scene Adaptive Optimization; specifically including: Stochastic scaling and data augmentation are introduced during the training phase of deep learning detection networks; Temperature compensation is achieved through color space conversion and brightness / contrast adjustment.

8. A target identification and damage assessment system based on the method of any one of claims 1 to 7, characterized in that, The system includes a data acquisition module, a multi-task processing module, a target matching module, a feature extraction module, and a decision evaluation module; The data acquisition module is used to acquire multimodal image data of the target before and after damage, construct a multi-source damage dataset, and perform preprocessing. The multi-task processing module is used to simultaneously perform target detection and preliminary damage assessment; The target matching module is used to achieve accurate correlation of the target ROI before and after damage; The feature extraction module is used to construct a multi-dimensional feature vector representing the damage state of the target; The decision evaluation module is used to output comprehensive damage assessment results by applying a cascaded classification mechanism.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the method as described in any one of claims 1 to 7 when it invokes the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images