A lightweight visual detection method and system for multimodal scarce samples based on modal reliability closed-loop control

Abstract

This invention relates to the field of visual inspection technology, and more particularly to a lightweight visual inspection method and system for multimodal scarce samples based on modal credibility closed-loop control. The method performs spatiotemporal registration and feature extraction on multimodal data, calculates modal credibility coefficients, and accordingly performs fusion path selection, pseudo-label screening, category prototype updating, and inference sub-path activation. This scheme can improve target detection accuracy, robustness, and edge deployment adaptability in scenarios with scarce labeled samples and complex environments.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to a lightweight visual inspection method and system for multimodal scarce samples based on modal reliability closed-loop control. Background Technology

[0002] Visual inspection technology is a crucial component of computer vision and intelligent sensing. Its goal is to classify, locate, and analyze the state of targets in images or videos, and it is now widely applied in robotics, autonomous driving, industrial defect detection, security monitoring, and intelligent manufacturing. With the development of deep learning technology, target detection methods based on convolutional neural networks are constantly evolving, driving continuous improvements in detection accuracy, speed, and application scope. Simultaneously, collaborative sensing using heterogeneous sensors in complex scenarios is gradually becoming an important direction for improving detection robustness.

[0003] However, current visual inspection technologies still heavily rely on large-scale, high-quality labeled data. For applications such as industrial sites, autonomous driving edge scenarios, nighttime adverse weather scenarios, and remote sensing of small targets, target samples often face challenges such as difficulty in acquisition, high labeling costs, uneven class distribution, and significant long-tail samples. Especially in real-world environments where new categories are constantly emerging and scenarios are continuously changing, traditional solutions relying solely on a single modality, sufficient labeling, and a static model structure often struggle to simultaneously achieve generalization ability, deployment efficiency, and detection stability. Existing technologies have recognized that target detection performance under limited sample conditions is significantly lower than conventional fully supervised methods, while multi-sensor fusion faces instability issues caused by noise, outliers, and scene fluctuations in dynamic environments.

[0004] From a multimodal detection perspective, to enhance adaptability to complex conditions such as occlusion, weak texture, low illumination, climate change, and distance variations, it is typically necessary to comprehensively utilize information from multiple modalities, including visible light, infrared, depth, lidar, or radar. Existing fusion methods can be broadly categorized into early fusion and late fusion: early fusion can obtain richer joint features at the input stage but is more sensitive to noise and outliers; late fusion is more robust to single-sensor noise but struggles to fully utilize cross-modal complementary information in intermediate layers. Therefore, in complex dynamic environments, schemes employing only fixed fusion levels and strategies often fail to adaptively adjust based on the current scene state, modal quality differences, and computational limitations, thus restricting the stability and efficiency of the model in practical deployments.

[0005] From the perspective of few-shot detection, the core objective of few-shot object detection is to enable the model to effectively identify and locate new categories of objects even with very few labeled samples. Existing research typically employs techniques such as meta-learning, prototype learning, and support set-query set feature aggregation. By constructing category prototypes, calculating similarity, or concatenating and enhancing feature maps, query features can learn the category information expressed by support features more quickly. Although these approaches alleviate the problem of insufficient new category samples to some extent, their detection performance depends not only on the feature aggregation method but also heavily on the quality of the prototype representation formed by the support features. Once the number of samples decreases, the scene background becomes more complex, or the quality of the input modality fluctuates, existing approaches may still experience problems such as decreased category recognition ability, insufficient localization accuracy, and unstable adaptation to new categories.

[0006] For example, existing technical document [CN116129226A] discloses a method and apparatus for few-sample target detection based on a multi-prototype hybrid module. This scheme extracts feature information from support images and query images through a backbone network, then processes the support features using the expectation-maximization algorithm to generate multiple prototype vectors with representative information. These prototype vectors are then aggregated with query features to activate query features related to the target, which are then fed into a subsequent detection module to complete target detection. Simultaneously, this scheme also introduces a contrastive training strategy to enhance the learning ability of new object categories. The advantage of this type of scheme is that it improves the detection performance of new object categories under few-sample conditions. However, based on its published content, its focus remains on the construction of support feature prototypes and the query feature aggregation mechanism, primarily addressing the problem of how to learn more representative category representations when samples are limited. CN116129226A does not provide an integrated control mechanism for the entire training and inference process to address issues such as spatiotemporal registration errors, noise level differences, data loss, and dynamic changes in modal contribution under multimodal input conditions. Therefore, its applicability still has room for improvement in practical applications where heterogeneous sensors are involved, scenarios change drastically, and computing power is limited.

[0007] For example, existing technical document [US20240062519A1] discloses a context-aware selective sensor fusion method for multi-sensor computing systems. This approach recognizes that most modern multi-sensor fusion methods typically perform fusion at a single location within the model and mostly employ static fusion algorithms independent of environmental context. Therefore, it proposes selectively performing early, late, or intermediate-layer fusion at different depths of the model based on scene context information, and dynamically switching between different fusion paths through a multi-branch architecture and gating strategies to improve perception robustness in complex driving scenarios while mitigating the computational burden on edge platforms. The advantages of this approach lie in its improvements to issues such as fixed fusion levels, insufficient robustness in complex scenarios, and high computational overhead in multi-branch systems. However, based on its published content, its technical focus is primarily on how to dynamically select the fusion method based on scene context, mainly targeting structural-level adaptation of the multi-sensor detection process, without further in-depth design for limited labeled samples, pseudo-label reliability control, category prototype memory evolution, and collaborative closed-loop control between the training and inference phases. Therefore, when the application scenario has multiple constraints such as modal heterogeneity, scarce annotations, dynamic environmental changes, and limited edge deployment, this paper still struggles to balance small-sample learning ability, pseudo-label quality assurance, and cross-modal detection stability.

[0008] In summary, while existing technologies have made some progress in few-shot target detection and context-aware multimodal fusion, they still have the following shortcomings: First, few-shot or semi-supervised detection schemes usually focus more on category representation enhancement, pseudo-label utilization, or prototype construction, lacking a fine characterization of the differences in multimodal data quality, making it difficult to uniformly evaluate and dynamically schedule the credibility of different modalities in complex scenarios; Second, although multimodal fusion schemes can improve perceptual robustness in complex environments to some extent, they often lean more towards scene-level or structure-level fusion path selection, and have not yet further integrated the modality quality evaluation results into pseudo-label screening, category prototype updating, detection sub-path activation, and lightweight inference control; Third, most existing schemes separate the training and inference phases, lacking a unified closed-loop control framework that can simultaneously address limited labeled samples, heterogeneous multimodal inputs, and edge deployment constraints.

[0009] Therefore, how to provide a new visual inspection method and system that can effectively align and evaluate the credibility of multimodal data under limited labeled samples, and jointly control the selection of fusion path, sample screening, category representation update and inference sub-path activation based on the credibility results, so as to ensure detection accuracy while taking into account robustness in complex scenarios and lightweight deployment capability, has become an urgent technical problem to be solved in this field. Summary of the Invention

[0010] The technical objective of this invention is to address the problems of existing visual inspection technologies in complex scenarios, such as scarce labeled samples, large fluctuations in multimodal data quality, fixed fusion strategies, insufficient reliability of pseudo-labels, and difficulty in balancing detection accuracy and computational efficiency under edge deployment conditions. This invention provides a lightweight visual inspection method, system, device, and storage medium based on modal credibility closed-loop control. This enables adaptive credibility assessment of multimodal input data and, based on this, collaboratively completes fusion path selection, pseudo-label screening, category prototype updating, and inference sub-path activation, thereby improving the accuracy, robustness, and deployment adaptability of target detection under scarce sample conditions.

[0011] Firstly, in order to achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0012] A lightweight visual detection method for multimodal scarce samples based on modal reliability closed-loop control includes the following steps:

[0013] S1. Obtain raw data of at least two modalities of the scene to be detected, and complete cross-modal spatiotemporal registration using a unified timestamp and calibration parameters to construct a labeled set containing a small number of labeled samples, an unlabeled set containing unlabeled samples, and an online sample set to be detected.

[0014] S2. Input the original data of the at least two modalities into the shared lightweight backbone network and the modality-specific feature encoding branch respectively, extract the multi-scale features of each modality, and calculate the modality confidence coefficient of each modality based on the alignment residual, noise index and effective information index between the multi-scale features of each modality and the reference alignment features.

[0015] S3. Based on the modality confidence coefficient of each modality and the scene complexity index of the current sample, select at least one fusion path from the preset early fusion path, mid fusion path and late fusion path to perform hierarchical feature fusion; when the modality confidence coefficient of any modality is lower than the first threshold, suppress the contribution of that modality to the fusion result or switch to the degraded detection path to obtain the fused feature representation.

[0016] S4. Construct a category prototype memory based on the labeled set, and use the teacher detection network to generate candidate boxes and pseudo-labels for the unlabeled set; perform joint screening on the pseudo-labels based on the pseudo-label confidence, cross-modal consistency score and modal confidence coefficient of each modality, and only use pseudo-labels that meet the reliability conditions for classification loss and localization loss calculation, and update the category prototype memory using samples that meet the reliability conditions.

[0017] S5. Input the fused feature representation, the category prototype memory, and the filtered pseudo-labels into the student detection network for training, and iteratively update the teacher detection network parameters based on the student detection network parameters; during the inference phase, dynamically activate the preset detection sub-path according to the scene complexity index and hardware resource index, and output the target category, target location, and detection confidence.

[0018] Preferably, in step S1, the at least two modal raw data include any two or more of visible light images, infrared images, depth images, lidar point clouds, and millimeter-wave radar echoes; the cross-modal spatiotemporal registration includes: completing frame-level alignment according to a unified timestamp, and mapping each modal data to a unified spatial coordinate system according to an intrinsic parameter matrix, extrinsic parameter matrix, projection matrix, or point cloud-to-image mapping relationship.

[0019] Preferably, in step S2, the shared lightweight backbone network is a depthwise separable convolutional network, a Ghost network, a lightweight Transformer network, or a combination thereof; the modality-specific feature encoding branches are used to extract texture features, thermal radiation features, geometric features, or reflection intensity features, respectively; the noise index is used to characterize the noise intensity, missing degree, or distortion degree of the current modality; and the effective information index is used to characterize the distinguishable texture, edge, geometric structure, or target response intensity of the current modality in the current scene.

[0020] Preferably, in step S2, the modal reliability coefficient Calculate using the following formula:

[0021] ;

[0022] In the formula, Indicates the first Modal reliability coefficients for each mode; Indicates the total number of modes; Indicates the first Alignment residuals for each mode; Indicates the first Noise index for each mode; Indicates the first Effective information indicators for each modality; Indicates the first Data missing rate for each modality; , , and These are weighting coefficients, and all are real numbers greater than 0.

[0023] Preferably, in step S3, the scene complexity index and the aforementioned hardware resource indicators Calculate according to the following formulas:

[0024] ;

[0025] ;

[0026] In the formula, Indicates the complexity of the scenario; Indicates the density of candidate targets; Indicates the target overlap rate; Indicates a blurred or obscured indicator; Indicators representing fluctuations in illumination or motion disturbances; , , , The weighting factor is the scene complexity.

[0027] Indicates hardware resource metrics; Indicates the percentage of remaining memory; Indicates the proportion of available computing power; Indicates the remaining power consumption budget percentage; , , The weighting factor is the hardware resource weighting factor.

[0028] when Activate the low-cost detection sub-path at the time; when and Time-activated precision enhancement detection subpath; when any mode of The degradation detection sub-path or single-modal backup path is activated at the appropriate time; among which... , , This is a preset threshold.

[0029] Preferably, in step S4, the category prototype memory uses an exponential sliding update method, and the category... prototype vector Update as follows:

[0030] ;

[0031] In the formula, Indicates the first The updated categories Prototype vector; Indicates the first The updated categories Prototype vector; This represents the prototype update smoothness coefficient, and ; This indicates that the product passed the reliability screening and belongs to the relevant category. The number of samples; Indicates the first Each belongs to category The fused feature vector of the samples.

[0032] Preferably, in step S4, the pseudo-tag reliability score Calculate using the following formula:

[0033] ;

[0034] In the formula, Indicates the first Reliability score of each pseudo-label; Indicates the first Confidence level of pseudo-label detection; Indicates the first Cross-modal consistency score of pseudo-labels; Indicates the first Modal reliability coefficients for each mode; Indicates the first The pseudo-label is in the first Modal support scores on each modality; Indicates the first The distance penalty term between the fused features of the sample corresponding to each pseudo-label and its class prototype; , , , These are weighting coefficients, and all are real numbers greater than 0;

[0035] when and At that time, the first A pseudo-label is used simultaneously for calculating both classification loss and localization loss; when the above conditions are not met, the pseudo-label is written to the uncertain sample buffer and does not participate in the current round of training; where... and This is a preset threshold.

[0036] Secondly, the present invention also provides a lightweight multimodal scarce sample visual detection system based on modal reliability closed-loop control, the system being used to implement the method, including:

[0037] The data acquisition and registration module is used to acquire raw data from at least two modalities and complete spatiotemporal registration.

[0038] The feature extraction module is used to extract multi-scale features of each modality using a shared lightweight backbone network and modality-specific feature encoding branches;

[0039] The modal credibility calculation module is used to calculate the modal credibility coefficient based on alignment residuals, noise index, effective information index, and data missing rate.

[0040] The fusion path selection module is used to select the fusion path and detection sub-path based on the modality confidence coefficient, scene complexity index, and hardware resource index.

[0041] The pseudo-label filtering and prototype update module is used to generate pseudo-labels, calculate pseudo-label reliability scores, and update the category prototype memory.

[0042] The teacher-student collaborative training module is used to train the student detection network and update the teacher detection network based on the fused feature representation, the category prototype memory, and the filtered pseudo-labels.

[0043] The results output module is used to output the target category, target location, and detection confidence.

[0044] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method described above.

[0045] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0046] The beneficial technical effects of this invention are as follows: By introducing a modal credibility closed-loop control mechanism, the alignment quality, noise level, effective information strength, and data loss of multimodal data after spatiotemporal registration are uniformly quantified, and this quantification result is applied throughout the entire process of fusion path selection, pseudo-label screening, category prototype updating, and inference sub-path activation. This enables the system to adaptively suppress the adverse effects of low-credibility modalities on detection results in complex scenarios, avoiding information redundancy, error propagation, and the masking of complementary information of weak modalities by strong modalities caused by fixed fusion strategies. This significantly improves the detection robustness under conditions such as occlusion, low illumination, blurring, and cross-scene changes. Simultaneously, by linking modal credibility with pseudo-labels... The combination of reliability and prototype memory updates can improve the utilization quality of unlabeled samples, reduce the accumulation of pseudo-label errors and the risk of class drift, and enhance the representation ability, classification accuracy and positioning accuracy of new class targets under scarce sample conditions. Furthermore, the present invention adopts a lightweight backbone network and dynamically activates the detection sub-path by combining scene complexity and hardware resource status, which can effectively reduce the number of model parameters, computational load and inference latency while ensuring detection performance, and improve the adaptability of edge device deployment. Therefore, the present invention can simultaneously take into account detection accuracy, environmental adaptability, training stability and real-time deployment capability, and is suitable for a variety of practical application scenarios such as industrial inspection, autonomous driving, remote sensing recognition and night monitoring. Attached Figure Description

[0047] Figure 1 This is a block diagram of the overall system structure of the present invention based on modal reliability closed-loop control.

[0048] Figure 2 This is a flowchart of the overall process of the method of the present invention.

[0049] Figure 3 This is a schematic diagram illustrating the multimodal feature extraction and modal credibility calculation of the present invention.

[0050] Figure 4 This is a schematic diagram illustrating the fusion path selection and degradation detection path switching of the present invention.

[0051] Figure 5 This is a schematic diagram illustrating the pseudo-label filtering and category prototype memory update of the present invention.

[0052] Figure 6 This is a schematic diagram illustrating the collaborative operation of the training and inference phases of this invention.

[0053] Figure 7 This is a comparison chart of the overall performance of different detection methods.

[0054] Figure 8 This is a comparison chart of recall rates under different complex operating conditions.

[0055] Figure 9 The curves showing the changes in pseudo-label accuracy and prototype drift distance during the training process.

[0056] Figure 10 A latency-power-accuracy balance diagram for edge deployment.

[0057] Figure 11 A visual comparison chart of typical test results. Detailed Implementation

[0058] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0059] The following combination Figures 1 to 11 The following describes specific embodiments of the present invention in further detail. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or conventional modifications made by those skilled in the art to the system structure, network layer configuration, threshold settings, training strategies, and deployment methods without departing from the concept of the present invention should all fall within the scope of protection of the present invention.

[0060] in, Figure 1This can be a block diagram of the overall system structure of the present invention; Figure 2 This can be a flowchart of the overall process of the present invention; Figure 3 This can be used as a schematic diagram for multimodal feature extraction and modal credibility calculation; Figure 4 This can be a schematic diagram of fusion path selection and downgraded detection path switching; Figure 5 This can serve as a schematic diagram for pseudo-label filtering and category prototype memory updates; Figure 6 This diagram illustrates the collaborative work between the training and inference phases. The following detailed implementation will be described in conjunction with the aforementioned diagram.

[0061] I. Terminology Explanation

[0062] To facilitate understanding of the technical solution of this invention, the following explanations are provided for the main terms used in this specification:

[0063] 1. Multimodal raw data

[0064] This refers to data derived from at least two different types of sensors or different imaging mechanisms, including but not limited to visible light images, infrared images, depth images, lidar point clouds, millimeter-wave radar echo maps, and their corresponding time-series frame data. This invention preferably uses visible light images and infrared images as examples for illustration, but is not limited thereto.

[0065] 2. Spatiotemporal registration

[0066] This refers to the process of uniformly aligning different modalities of data in both the temporal and spatial dimensions. Temporal alignment ensures that different modalities correspond to the same time point or the same sampling window; spatial alignment ensures that different modalities are comparable within a unified reference coordinate system.

[0067] 3. Modal reliability coefficient

[0068] This refers to a quantitative indicator used to characterize the trustworthiness of a particular modality in the current sample, current scene, and current time slice. This coefficient comprehensively considers the alignment residual, noise level, effective information strength, and data missingness of the modality, and participates in subsequent fusion path selection, pseudo-label screening, prototype update, and inference path activation.

[0069] 4. Scene complexity metrics

[0070] It refers to a comprehensive index that reflects the target density, target overlap, blur or occlusion degree, and lighting or motion disturbance degree in the current scene to be detected, and is used to determine whether a higher precision detection path needs to be called.

[0071] 5. Hardware resource indicators

[0072] This refers to indicators that characterize the current state of available computing resources on a device, including but not limited to the proportion of remaining memory, the proportion of available computing power, and the proportion of remaining power consumption budget. These indicators are used to constrain the selection of detection paths in order to balance detection accuracy and real-time deployment capabilities.

[0073] 6. Fusion Path

[0074] This refers to the path methods for information fusion of multimodal features, including early fusion path, mid-fusion path, and late fusion path. The early fusion path focuses on the joint function of low-level features, the mid-fusion path focuses on the semantic interaction of mid-to-high-level features, and the late fusion path focuses on the fusion of results after independent prediction of each modality.

[0075] 7. Category Prototype Memory

[0076] This refers to a data structure that stores representative feature vectors of each target category. It is used to enhance the category representation capability under scarce sample conditions and participates in the determination of the reliability of pseudo-labels and the training of student detection networks.

[0077] 8. False Tag Reliability Score

[0078] This refers to the score used to evaluate the quality of candidate labels generated by the teacher detection network on unlabeled samples. The score takes into account detection confidence, cross-modal consistency score, modal support score, and distance penalty between the sample and the class prototype.

[0079] 9. Degradation detection sub-path

[0080] This refers to the system automatically switching to a low-cost or single-primary-mode-priority detection path when a certain modality has low confidence or hardware resources are limited, in order to suppress the adverse effects of low-confidence modalities on the detection results.

[0081] 10. Collaborative training between teachers and students

[0082] This refers to a training mechanism that uses a teacher detection network to generate pseudo-labels for unlabeled samples, and then a student detection network learns under the constraints of these pseudo-labels, while updating the parameters of the teacher detection network through an exponential moving average.

[0083] II. System Structure

[0084] like Figure 1 As shown, the present invention provides a lightweight multimodal scarce sample visual detection system based on modal credibility closed-loop control, which includes at least a data acquisition and registration module, a feature extraction module, a modal credibility calculation module, a fusion path selection module, a pseudo-label screening and prototype update module, a teacher-student collaborative training module, a result output module, and a software / hardware operating platform for carrying the above modules.

[0085] (I) Data Acquisition and Registration Module

[0086] This module is used to acquire raw data from at least two modalities and map them to a unified temporal and spatial reference. In a preferred embodiment, a visible light industrial camera and a long-wave infrared camera are used to simultaneously acquire images via hardware triggering; for trimodal scenes, a depth camera or LiDAR can also be connected in parallel. The output of this module is a registered multimodal sample pair for use by subsequent modules.

[0087] (II) Feature Extraction Module

[0088] This module consists of a shared lightweight backbone network and modality-specific feature encoding branches. The shared lightweight backbone network ensures that the overall model parameter count and computational cost are controllable; the modality-specific feature encoding branches compensate for the differences in texture, thermal information, and geometric information representation among different modalities. This module outputs feature maps of each modality at multiple scales.

[0089] (III) Modal Reliability Calculation Module

[0090] This module calculates the corresponding modal reliability coefficient based on the differences between each modal feature and the reference alignment feature, the noise level, the strength of effective information, and the proportion of invalid data. The obtained reliability coefficient is not only used for fusion control of the current sample, but also preferably for temporal smoothing control in continuous frame scenarios.

[0091] (iv) Fusion Path Selection Module

[0092] This module selects between early, mid, and late fusion paths based on modal reliability coefficients, scene complexity indicators, and hardware resource indicators, and triggers a downgraded detection sub-path when necessary. This module is one of the key components of this invention, and its function is to achieve adaptive detection based on samples, scenes, and resources.

[0093] (v) Pseudo-tag filtering and prototype update module

[0094] After the teacher detects the candidate bounding boxes and candidate categories output by the network, this module calculates a reliability score for the pseudo-labels and selects pseudo-labels that can be used for training based on a reliability threshold. Simultaneously, this module updates the category prototype memory using the selected reliable samples to continuously enhance the category representation capability.

[0095] (vi) Teacher-student collaborative training module

[0096] This module is used to execute the training process. A semi-supervised training method is preferred, where the teacher network processes weakly augmented samples and the student network processes strongly augmented samples; the teacher network parameters are obtained from the student network parameters through an exponential moving average. This module also undertakes the joint optimization of supervised loss, unsupervised loss, prototype constraint loss, consistency loss, and path selection regularization term.

[0097] (vii) Results Output Module

[0098] This module outputs the final target category, target location, and detection confidence. In video scenarios, it can further output the detection results after cross-frame stabilization; in industrial inspection scenarios, it can also output the defect category, defect bounding box, and alarm flag; in edge device scenarios, it can also output the currently selected path, current modality confidence, and inference time.

[0099] (viii) Software / hardware operating platform

[0100] This invention can be deployed on servers, industrial edge computing boxes, in-vehicle computing platforms, or embedded AI terminals. In preferred embodiments, it can be implemented using processors with GPU or NPU acceleration capabilities, volatile memory, non-volatile memory, and sensor interface buses. Software implementation can employ PyTorch, TensorRT, or other deep learning inference frameworks.

[0101] III. Specific Technical Route for Implementing the Method of the Invention

[0102] like Figure 1 As shown, a lightweight visual detection method for multimodal scarce samples based on modal reliability closed-loop control includes the following steps:

[0103] S1. Obtain raw data of at least two modalities of the scene to be detected, and complete cross-modal spatiotemporal registration using a unified timestamp and calibration parameters to construct a labeled set containing a small number of labeled samples, an unlabeled set containing unlabeled samples, and an online sample set to be detected.

[0104] S2. Input the original data of the at least two modalities into the shared lightweight backbone network and the modality-specific feature encoding branch respectively, extract the multi-scale features of each modality, and calculate the modality confidence coefficient of each modality based on the alignment residual, noise index and effective information index between the multi-scale features of each modality and the reference alignment features.

[0105] S3. Based on the modality confidence coefficient of each modality and the scene complexity index of the current sample, select at least one fusion path from the preset early fusion path, mid fusion path and late fusion path to perform hierarchical feature fusion; when the modality confidence coefficient of any modality is lower than the first threshold, suppress the contribution of that modality to the fusion result or switch to the degraded detection path to obtain the fused feature representation.

[0106] S4. Construct a category prototype memory based on the labeled set, and use the teacher detection network to generate candidate boxes and pseudo-labels for the unlabeled set; perform joint screening on the pseudo-labels based on the pseudo-label confidence, cross-modal consistency score and modal confidence coefficient of each modality, and only use pseudo-labels that meet the reliability conditions for classification loss and localization loss calculation, and update the category prototype memory using samples that meet the reliability conditions.

[0107] S5. Input the fused feature representation, the category prototype memory, and the filtered pseudo-labels into the student detection network for training, and iteratively update the teacher detection network parameters based on the student detection network parameters; during the inference phase, dynamically activate the preset detection sub-path according to the scene complexity index and hardware resource index, and output the target category, target location, and detection confidence.

[0108] (a) Step S1: Multimodal data acquisition, registration and dataset construction

[0109] like Figure 2 As shown, step S1 includes multimodal data acquisition, time synchronization, spatial calibration, data preprocessing, and the construction of labeled / unlabeled / online sample sets to be detected.

[0110] 1. Multimodal data acquisition

[0111] In a preferred embodiment, the data acquisition target is a target detection task in a complex environment, and a dual-mode acquisition unit consisting of a visible light camera and an infrared camera is selected. The output resolution of the visible light camera is preferably... or The frame rate is preferably 25fps or 30fps; the output resolution of the infrared camera is preferably... or The frame rate is preferably the same as that of the visible light camera. In the case of a three-modal embodiment, a depth camera or LiDAR can also be added, wherein the LiDAR data can be first transformed into a bird's-eye view feature map or projected into a dense depth map before participating in subsequent processing.

[0112] 2. Time synchronization

[0113] The preferred methods for time alignment of multimodal data are hardware-triggered synchronization, PTP (Precise Time Protocol) synchronization, or a unified timestamp buffer queue. In this embodiment, a timestamp difference of less than 10ms is used as the pairing criterion; if the timestamp difference is greater than the threshold, the sample set is discarded or the synchronization frame is reconstructed using interpolation.

[0114] 3. Spatial calibration and registration

[0115] For calibration between two 2D image modes, a planar calibration plate can be used to calibrate intrinsic and extrinsic parameters; for calibration between a 2D image mode and a 3D point cloud mode, a spatial calibration target can be used to map the coordinate system. For general projection relationships, the following mapping formula can be used:

[0116] ;

[0117] in, Represents the homogeneous pixel coordinates of the target point projected onto the visible light image plane; This represents the intrinsic parameter matrix of a visible light camera; Indicates by the first Rotation matrix from each modal coordinate system to the visible light camera coordinate system; Indicates the first Three-dimensional point coordinates in each mode; Indicates by the first Translation vector from the modal coordinate system to the visible light camera coordinate system.

[0118] In a preferred embodiment of the dual-modal two-dimensional image, spatial alignment from the infrared image to the visible light image is achieved using a homography matrix or sparse control point fitting method based on feature point matching. After spatial alignment, the different modalities are uniformly scaled to the input size, for example... .

[0119] 4. Data Preprocessing

[0120] For visible light images, normalization, color space enhancement, brightness and contrast perturbation, random flipping, and random scaling are performed. For infrared images, temperature range cropping, contrast-limited adaptive histogram equalization, pseudo-color mapping, or single-channel normalization are performed. For depth maps or point clouds, invalid value filling, outlier removal, voxelization, or depth normalization can be performed. It is worth noting that geometric transformations must be performed synchronously across modalities to ensure that the bounding boxes of the same target remain consistent across multiple modalities.

[0121] 5. Dataset Construction

[0122] To enable those skilled in the art to implement this invention directly, the present invention preferably employs a data organization method of basic class + new category + unlabeled sample pool. Taking a dual-modal detection scenario as an example, the following dataset can be constructed:

[0123] (1) Basic class annotation set For example, it includes 8 basic categories, with 300 to 1000 pairs of labeled visible-infrared samples for each category;

[0124] (2) New category annotation set For example, it includes 4 to 8 new categories, with each category retaining only 1, 3, 5, or 10 pairs of labeled samples to simulate scarce sample conditions;

[0125] (3) Unlabeled sample set For example, it may contain 10,000 to 50,000 pairs of unlabeled samples, covering various scenarios where basic and new categories appear;

[0126] (4) Online sample set to be tested Used for actual inference deployment.

[0127] In a preferred embodiment, a 5-shot new category setting is preferred, meaning that only 5 pairs of labeled samples are used for each new category, while a large number of unlabeled samples are retained for training. This not only demonstrates the value of the invention in scenarios with scarce samples, but also facilitates the enhancement of detection performance through pseudo-labels and prototype memory.

[0128] (ii) Step S2: Construction of shared lightweight backbone network, modality-specific branches and modality credibility closed loop

[0129] Step S2 is one of the core steps of this invention. For example... Figure 3 As shown, this step not only involves multimodal feature extraction, but also constructs a modal credibility coefficient that runs through the entire subsequent training and inference process, which is one of the key differences between this invention and the prior art.

[0130] 1. Input Definition and Overall Structure

[0131] Taking the dual-modal implementation as an example, the input visible light image is denoted as... The input infrared image is denoted as If the infrared image is single-channel, it first passes through a... The convolutional layers are mapped to 16-channel shallow features, which are then interfaced with a shared lightweight backbone network.

[0132] The shared lightweight backbone network preferably adopts a hybrid structure of Ghost modules, depthwise separable convolutional modules, and lightweight Transformer modules. For ease of implementation, a set of directly reproducible preferred network structures are given below:

[0133] (1) Stem layer: two cascaded layers Convolutional blocks with strides of 2 and 2, and output channels of 32 and 64 respectively;

[0134] (2) Stage 1: 2 Ghostbottleneck or C2f lightweight residual units, with 64 output channels;

[0135] (3) Stage 2: 3 lightweight residual units, 128 output channels, output scale denoted as Spatial resolution is ;

[0136] (4) Stage 3: 3 lightweight residual units, 256 output channels, output scale denoted as Spatial resolution is ;

[0137] (5) Stage 4: Two lightweight Transformer or depthwise separable convolutional enhancement units, with 512 output channels and an output scale denoted as . Spatial resolution is .

[0138] In this embodiment, the parameters of the shared lightweight backbone network are shared between different modalities to reduce the number of model parameters; however, modality-specific feature encoding branches are set after the output of each scale to compensate for modal differences.

[0139] 2. Modality-specific feature encoding branch

[0140] For visible light modes, the preferred mode-specific branch includes Channel-adjusted convolutions, edge-enhancing convolutions, and lightweight channel attention modules are used to highlight texture, edge, and color contrast information. For the infrared mode, the mode-specific branch preferably includes local thermal response enhancement convolutions, background suppression convolutions, and lightweight spatial attention modules to improve the saliency of thermal targets. Each mode-specific branch outputs a feature map with the same scale as the shared backbone, denoted as . , , .

[0141] To ensure compatibility in subsequent fusion, it is preferable to uniformly adopt the same approach for features at all scales across different modal branches. Convolution maps to the same channel dimension, for example in , , The channels are uniformly set to 128, 256, and 256 respectively.

[0142] 3. Determination of reference alignment features

[0143] In the dual-modal visible-infrared embodiment, the visible mode is preferably used as the reference mode, and its scale is denoted as [missing information]. The following features are Infrared modal characteristics are denoted as For three-modal or more modal implementations, the mode with the highest resolution and best positioning stability can be selected as the reference mode, or a reference feature template before fusion can be used as a unified reference.

[0144] 4. Calculation of Alignment Residuals

[0145] In order to quantify the first The degree of consistency between the first mode and the reference mode is first determined based on the calibration parameters or feature matching results. Geometric transformation parameters of each mode relative to the reference mode Then apply it to the first Each modal feature map. The alignment residual is preferably calculated using the following method:

[0146]

[0147] in, Indicates the first Alignment residuals for each mode; This represents the total number of valid feature points or valid pixel locations involved in the calculation. Indicates the use of geometric transformation parameters For the Modal feature map The result after performing space alignment; Represents the reference modal feature map; express Norm; This represents the geometric error term obtained based on keypoint matching or bounding box reprojection. , This represents the residual weighting coefficient, with preferred values ​​ranging from 0.5 to 0.9 and from 0.1 to 0.5, respectively.

[0148] 5. Extraction of noise indicators, effective information indicators, and data missing rate.

[0149] To comprehensively evaluate the quality of a particular mode in the current sample, relying solely on alignment residuals is insufficient. Therefore, this invention further constructs a noise index. Effective information indicators and data missing rate .

[0150] Among them, noise index It can be obtained from one or more linear combinations of the following: ambiguity score, proportion of bad pixels, proportion of saturated regions, proportion of low signal-to-noise ratio regions, and proportion of hot noise. Preferably, these sub-indicators are normalized to... Weighted summation of intervals.

[0151] Effective Information Indicators This is used to measure the information strength of the modality that is truly valuable for target detection. It is preferably composed of local information entropy, edge density, foreground response strength, saliency response, and activation energy associated with the target region. This contrasts with noise metrics. The larger the value, the more helpful the mode is for detection.

[0152] Data missing rate This indicates the proportion of invalid data in the total data for that modality. For example, a large number of invalid depth points in a depth camera, large saturated areas in an infrared image, or distant sparse areas in a point cloud can all reflect a high invalidity. It is preferable to calculate directly using invalid pixel count or the proportion of low-confidence pixels.

[0153] 6. Calculation of Modal Reliability Coefficient

[0154] In obtaining , , and Then, the modal reliability coefficient is calculated using the following formula. :

[0155] ;

[0156] in, Indicates the first Modal reliability coefficients for each mode; Indicates the total number of modes; Indicates the first Alignment residuals for each mode; Indicates the first Noise index for each mode; Indicates the first Effective information indicators for each modality; Indicates the first Data missing rate for each modality; , , , These are weighting coefficients. Preferably, Take a value of 0.8 to 1.5. Take a value of 0.5 to 1.2. Take a value of 0.8 to 1.6. Take a value of 0.5 to 1.0. In one specific embodiment, it can be taken as... , , , .

[0157] Since the statistical distribution may vary between different samples, it is preferable to perform calculations beforehand. , , and First, perform sliding window normalization or minimum-maximum normalization to ensure that the above indicators are comparable across different batches.

[0158] 7. Temporal smoothing of modal reliability

[0159] In video stream or continuous frame detection scenarios, to avoid drastic fluctuations in modal confidence between adjacent frames, a first-order exponential smoothing method can be used to update the smoothed modal confidence:

[0160] ;

[0161] in, Indicates the first Frame time Smoothness reliability of each modality; Indicates the first The smoothness reliability of this modality at frame rate; Indicates the first Modal reliability calculated in real-time within a frame; This represents the smoothing coefficient, with a preferred value of 0.6 to 0.9.

[0162] Through the above design, the present invention does not only introduce modal quality evaluation in the fusion stage, but also establishes a unified and reliable metric that can be repeatedly invoked. This reliable metric will continue to participate in the control in subsequent steps S3, S4 and S5, forming a closed loop.

[0163]

[0164] (III) Step S3: Scene complexity awareness fusion path selection and degradation detection control

[0165] Step S3 is another core step of this invention. For example... Figure 4 As shown, this invention does not employ a fixed multimodal fusion method, but rather makes a dynamic selection among early fusion, mid-fusion, late fusion, and degraded detection sub-paths based on modal reliability, scene complexity, and hardware resource status, thereby balancing the accuracy requirements of complex scenes with the real-time requirements of edge detection.

[0166] 1. Construction of Scene Complexity Metrics

[0167] This invention uses the following formula to define the scene complexity index. :

[0168] ;

[0169] in, Indicates the complexity of the scenario; The candidate target density can be obtained by dividing the number of coarsely detected candidates in the current frame by the image area; The target overlap ratio can be obtained from the average intersection-union ratio among the foreground candidate boxes; The ambiguity or occlusion index can be calculated by combining the Laplacian ambiguity and the occlusion estimation results; Indicators representing light fluctuations or motion disturbances, such as those obtained from the variance of brightness changes, exposure instability, or the degree of optical flow disturbance; , , , The weighted coefficients for scene complexity are used to select the best option that satisfies... In one embodiment, it is preferable to... , , , .

[0170] 2. Construction of Hardware Resource Indicators

[0171] This invention uses the following formula to define hardware resource indicators. :

[0172] ;

[0173] in, Indicates hardware resource metrics; Indicates the percentage of remaining memory; This indicates the proportion of available computing power, which can be calculated by reverse conversion from the current GPU / NPU idle rate or CPU load. Indicates the remaining power consumption budget percentage; , , The resource weighting coefficient is used to select the optimal one that satisfies... In one embodiment, it is preferable to... , , .

[0174] 3. Calculation of routing feature vectors and path weights

[0175] In the preferred implementation, the global pooling result of multimodal features, the smoothed modality confidence, the scene complexity index, and the hardware resource index are concatenated into a routing feature vector. The weights of each fusion path are then output via a multilayer perceptron:

[0176] ;

[0177] in, Represents the routing feature vector; Indicates the first The result of performing global average pooling on each modality feature map; This represents a vector concatenation operation; Indicates the modal confidence level after smoothing; Indicates the complexity of the scenario; This indicates hardware resource metrics.

[0178] Furthermore, the path weights for early, mid, and late fusion paths are calculated in the following form:

[0179] ;

[0180] in, , , These represent the weights of the early, mid, and late fusion paths, respectively. Represents a multilayer perceptron; This represents the normalization function.

[0181] In one embodiment, a hard routing method that selects the path with the highest weight can be used; in another embodiment, a soft routing method that uses weighted parallel connections can also be used. To improve deployment stability, a two-level control system of primary path + secondary path is preferred, that is, the path with the highest weight is selected as the primary path, and the path with the second highest weight is selected as the backup path. Switching to the backup path is only performed when the primary path malfunctions or the reliability of the current frame changes abruptly.

[0182] 4. Implementation of the early fusion path

[0183] When the system determines that the current scenario is relatively simple, hardware resources are limited, but the credibility differences between various modalities are not significant, the early fusion path can be activated. This path preferably fuses information from different modalities at the low-level feature stage, for example, in... Alternatively, perform feature concatenation or confidence-weighted summation at a shallower level:

[0184] ;

[0185] in, Indicates early fusion output features; Indicates the first The credibility coefficient of each modality; Indicates the first Channel alignment mapping function for low-level features of each modality; Indicates the first Low-level feature maps of each modality.

[0186] Early fusion paths have lower computational costs and are suitable for real-time inference on edge devices, but they are more sensitive to low-confidence modalities. Therefore, this path is not prioritized when the confidence level of any modality is too low.

[0187] 5. Implementation of the integration path

[0188] When the scenario is complex and the device has sufficient resources, the mid-level convergence path is preferred. The mid-level convergence path... , or Introducing cross-modal interaction at mid-to-high-level feature stages can utilize complementary multimodal information while avoiding the direct intrusion of low-level noise. It is preferable to perform credibility-weighted fusion and lightweight cross-attention at each scale.

[0189] ;

[0190] in, Representing scale The characteristics of integration in the middle; Indicates the first The credibility coefficient of each modality; This represents the channel alignment mapping function; Indicates the first Each modality at scale The following feature map; Indicates the first The modality and the first Each modality at scale Interaction weights under; This represents lightweight cross-attention operations.

[0191] In one specific embodiment, Low-rank attention or channel grouping attention can be used to control the computational load; It can be obtained by normalizing the product of the credibility of each modality, thereby automatically reducing the cross-modal interaction strength when the credibility of one side is low.

[0192] 6. Implementation of the late fusion path

[0193] When certain modalities are difficult to align precisely in space, or when individual predictions for certain modalities are relatively stable, a late fusion path can be chosen. The late fusion path performs detection head prediction independently for each modality, and then performs a weighted fusion of the final classification score and bounding box results. The final bounding box results are preferably output in the following format:

[0194] ;

[0195] in, This represents the final bounding box after merging; Indicates the first The credibility coefficient of each modality; Indicates the first The classification confidence of each modality corresponds to a candidate bounding box; Indicates the first Each modality independently predicts its bounding box.

[0196] The late fusion path is suitable for handling situations where the alignment accuracy of each modality is low but the detection quality of a single modality is good. It is also easier to directly reduce the weight of a certain modality's result when a certain modality is severely contaminated.

[0197] 7. Triggering mechanism for degradation detection sub-paths

[0198] When the confidence level of a certain modality falls below a first threshold, the system should suppress the contribution of that modality to the fusion result, or switch to a degraded detection subpath. In one embodiment, a configuration is set... When any mode When the feature weight of the branch corresponding to the modality is set to 0 or minimum value, only the main modality is retained for detection. When both modalities are at medium confidence but resources are insufficient, the main modality + lightweight compensation mode is preferred, that is, the main modality is fully forward-looking, and the auxiliary modality only participates in one-time compensation at key scales.

[0199] 8. Path switching anti-shake

[0200] To avoid latency fluctuations caused by frequent path switching in continuous frame scenarios, this invention preferably introduces a switching anti-shake mechanism. Specifically, when a new path needs to appear continuously... The actual switching only occurs when the frame is above a certain level, among which... The optimal frame rate is 3 to 5 frames. This maintains routing sensitivity while reducing operational jitter.

[0201] Through step S3 above, the present invention incorporates modal reliability, scenario complexity, and hardware resources into a unified routing decision, rather than performing detection solely according to a fixed network structure, thereby significantly enhancing the system's adaptability to complex scenarios and scenarios with limited resources.

[0202] (iv) Step S4: Construction of category prototype memory, screening of pseudo-label reliability and teacher-student collaborative training

[0203] Step S4 is one of the core training steps of this invention, such as... Figure 5 and Figure 6 As shown, this invention achieves stable performance improvement under scarce sample conditions through the linkage of three elements: category prototype memory, modality credibility-based pseudo-label screening, and teacher-student collaborative updating.

[0204] 1. Initialization of the category prototype memory

[0205] For each category Foreground region features are extracted from labeled samples and used as the prototype initialization source for this category. Specifically, RoIAlign is first performed on the fused feature map using the labeled bounding boxes to obtain foreground region features of uniform size, and then global average pooling and linear mapping are used to obtain the category feature vector. The initial prototype vector is obtained by averaging the features of multiple samples of the same category. .

[0206] In one embodiment, the category prototype dimension is preferably 256 dimensions; if a multi-scale prototype is used, then... , , Each of the three scales stores a set of prototypes, which are then combined in a weighted manner during matching.

[0207] 2. Category Prototype Update

[0208] During training, once a pseudo-labeled sample passes the reliability screening, its fused features can be used to update the prototype of the corresponding category. Exponential sliding update is preferred.

[0209] ;

[0210] in, Indicates the first The updated categories Prototype vector; Indicates the first The updated categories Prototype vector; This represents the prototype update smoothing coefficient, with a value between 0 and 1, preferably between 0.85 and 0.98. This indicates that this round has been determined to be reliable and belongs to the category. The number of samples; Indicates the first Each belongs to category The fusion feature vector corresponding to the sample.

[0211] When updating the prototype, it is preferable to set a category capacity balancing strategy to prevent high-frequency categories from excessively dominating the prototype evolution. Specifically, this can be achieved by setting a higher update weight for low-frequency classes or by limiting the number of samples written to each class in a single round.

[0212] 3. Organization methods of teacher networks and student networks

[0213] Preferably, the teacher detection network and the student detection network have the same network topology, including a shared lightweight backbone network, modality-specific branches, confidence calculation units, routing decision units, and detection heads. During training, a weakly augmented version of unlabeled samples is input into the teacher network, and a strongly augmented version is input into the student network. The teacher network outputs candidate boxes, classification distributions, and path selection results, while the student network learns under pseudo-label constraints.

[0214] The teacher network parameters are updated using an exponential moving average based on the student network parameters, with the following update formula:

[0215] ;

[0216] in, Indicates the first Teacher network parameters after the next iteration; This indicates the network parameters for the previous round of teacher training; This indicates the current student's network parameters; This represents the coefficient of the exponential moving average, with a preferred value of 0.99 to 0.9995.

[0217] 4. Teachers output candidate pseudo-tags online.

[0218] The teacher network outputs a set of candidate boxes for unlabeled samples. , corresponding category confidence and category distribution vector Each candidate pseudo-label must contain at least the following information: category Location box Confidence level , corresponding fusion feature vector Multimodal support status and current path type.

[0219] To make the pseudo-labels more reliable, it is preferable to first use a basic confidence threshold for filtering. For example, when Discard directly when; Then proceed to the next reliability scoring process.

[0220] 5. Calculation of cross-modal consistency score

[0221] The reliability of pseudo-labels depends not only on the single-classification confidence of the teacher network, but also on the degree of common support for the target across different modalities. This invention preferably calculates cross-modal consistency scores. In a dual-modal embodiment, the following form may be adopted:

[0222] l

[0223] in, Indicates the first Cross-modal consistency score of pseudo-labels; and These represent the bounding boxes predicted by the visible light mode and the infrared mode for the same candidate target, respectively. Indicates intersection, union, and ratio; and These represent the category distribution vectors corresponding to the two modes, respectively; Indicates the Jensen-Shannon divergence; This represents the fused feature vector of the candidate target; This represents the prototype vector of the candidate category; Indicates cosine similarity; , , This represents the consistency weighting coefficient, preferably satisfying the condition that the sum of the three factors is 1.

[0224] 6. Modal support score and distance penalty

[0225] No. The pseudo-label is in the first Modal support score on each modality The preferred definition is the normalized result of the difference between the foreground response and the background response within the pseudo-label bounding box region of the modality. The larger the value, the more the modality supports the pseudo-tag.

[0226] Distance penalty This is used to reflect the degree of deviation between sample features and the category prototype. The following form is preferred:

[0227] ;

[0228] in, Indicates the first The prototype distance penalty term for each pseudo-labeled sample; This represents the fusion feature vector of the sample; The prototype vector representing the category corresponding to the sample; This represents the cosine similarity. The smaller the value, the more closely the sample conforms to the current category's prototype distribution.

[0229] 7. False Tag Reliability Score

[0230] After considering confidence level, cross-modal consistency, modal support, and prototype deviation, the pseudo-label reliability score is calculated using the following formula:

[0231] ;

[0232] in, Indicates the first Reliability score of each pseudo-label; Indicates the first Confidence level of pseudo-label detection; Indicates the cross-modal consistency score; Indicates the first Modal reliability coefficients for each mode; Indicates the first The pseudo-label is in the first Modal support scores on each modality; Indicates the prototype distance penalty term; , , , These are weighting coefficients. Preferably, Take a value of 0.25 to 0.40. Take a value of 0.20 to 0.35. Take a value of 0.20 to 0.35. Take a value of 0.10 to 0.20.

[0233] 8. Three-level pseudo-label filtering strategy

[0234] To further improve training stability, pseudo-labels are preferably classified into three categories: highly reliable pseudo-labels, moderately reliable pseudo-labels, and low-reliability pseudo-labels.

[0235] (1) When and At that time, it is defined as a strongly reliable pseudo-label, which participates in the calculation of classification loss and localization loss, and also participates in the category prototype update;

[0236] (2) When and When defined as a reliable pseudo-label, it only participates in classification loss, consistency loss, and part of prototype alignment loss, and does not participate in bounding box regression update;

[0237] (3) When or When a false label is defined as low reliability, it is written into the uncertain sample buffer and does not participate in the training in the current round, but is only used for evaluation in subsequent iterations.

[0238] The advantage of this tiered screening method is that it avoids wasting potentially effective samples due to overly strict thresholds, and also avoids allowing erroneous pseudo-labels to pollute the training process on a large scale due to overly lenient thresholds.

[0239] 9. Student Network Loss Function

[0240] In student network training, the optimal method is to jointly optimize the supervised loss, unsupervised loss, prototype constraint loss, consistency loss, and routing regularization term. The total loss function can be written as:

[0241] ;

[0242] in, Indicates the total loss; This indicates supervised loss on labeled samples; This represents the unsupervised loss based on pseudo-labels; Represents the category prototype constraint loss; This indicates a loss of consistency. This indicates a regular expression for path selection; , , , This represents the corresponding loss weight, which is preferably set to 1.0, 0.2-0.5, 0.1-0.3, and 0.05-0.20, respectively.

[0243] The monitoring loss can be further defined as:

[0244] ;

[0245] in, Indicates supervised classification loss; Indicates monitoring and positioning loss; This represents the distribution focus loss or boundary distribution loss; , This represents the weighting coefficient.

[0246] Unsupervised loss optimization employs full updates for highly reliable pseudo-labels and reduced-weight updates for moderately reliable pseudo-labels. This can be written as:

[0247] ;

[0248] in, This represents a set of strongly reliable pseudo-labels; Represents a set of reliable pseudo-labels; The weighting coefficient for reliable pseudo-labels is represented by the value of [value]. .

[0249] The prototype constraint loss optimizes the foreground features to be close to the prototype of its own category and far away from prototypes of other categories, which can be achieved by using contrast loss or center loss; the consistency loss optimizes the consistency between teachers and students in path output, category distribution and bounding box prediction; the path regularization term is used to suppress frequent router oscillations.

[0250] Through the above step S4, the present invention integrates modality credibility, prototype memory and pseudo-label quality control under scarce sample conditions to form a complete training closed loop, thereby significantly improving the detection feasibility and stability in scenarios with small samples or few labels.

[0251] (v) Step S5: Dynamic reasoning, result output and deployment control

[0252] Step S5 corresponds to the online inference process during actual deployment. For example... Figure 6 As shown, after the sample to be detected is input into the system, the same preprocessing, feature extraction and modality confidence calculation as in the training phase are performed first, and then the corresponding sub-path is activated according to the scene complexity and hardware resources.

[0253] 1. Reasoning process

[0254] The execution order of the online inference phase can be as follows: Input the online sample to be detected. Data registration Feature extraction Modal reliability calculation Routing decision Path execution Detection head output Results fusion Nonmaximum suppression Output the detection results.

[0255] 2. Detect subpath activation rules

[0256] The following rules are preferred:

[0257] when and At that time, activate the low-cost detection sub-path;

[0258] when and At that time, activate the precision enhancement detection sub-path;

[0259] When any mode When this occurs, activate the downgraded detection sub-path or the single-modal backup path.

[0260] in, , , These represent the scene complexity threshold, hardware resource threshold, and modal confidence threshold, respectively. In one embodiment, it is preferred to take... , , .

[0261] 3. Output Results

[0262] The final output includes the target category, bounding box location, and detection confidence score. When performing non-maximum suppression on the bounding box, a confidence threshold of 0.25 and an NMS threshold of 0.50–0.65 are preferred. For industrial inspection scenarios, the results can also be mapped to defect alarm signals; for security or automotive scenarios, cross-frame correlation can be performed in conjunction with a tracking module.

[0263] IV. Structured Disclosure of Model Structure, Training Methods, Parameter Selection, and Dataset Usage

[0264] To ensure that those skilled in the art can directly implement this invention without creative effort, the model structure, training method, parameter selection, and dataset usage are hereby disclosed.

[0265] (a) Model Structure Disclosure

[0266] Input size is preferred ;

[0267] The backbone network preferably adopts a hybrid structure combining lightweight convolutions and lightweight Transformers;

[0268] The detection head can be an anchor-free head or a decoupled detection head with distribution focus regression;

[0269] The router preferably adopts a 2-layer fully connected structure, and the hidden layer dimension is preferably 128 or 256;

[0270] The preferred dimension of the category prototype vector is 128 or 256.

[0271] For dual-modal scenarios, it is possible to , , Three scales are involved in routing and fusion;

[0272] For a three-modal scenario, you only need to add a corresponding dedicated coding branch for the new modality and extend the modality dimension in the credibility calculation and routing.

[0273] (II) Disclosure of Training Methods

[0274] The training phase is divided into two stages:

[0275] The first stage is a supervised warm-up stage, which uses only the labeled set for training for 10 to 30 epochs.

[0276] The second stage is the semi-supervised joint training stage, which uses labeled and unlabeled sets for joint training for 50 to 150 epochs.

[0277] The AdamW optimizer is preferred, and the initial learning rate can be set to [value missing]. to The weight decay can be set to 0.01 to 0.05; the SGD optimizer can also be used.

[0278] The preferred approach is to use a cosine learning rate decay strategy and a warm-up of 3 to 5 epochs.

[0279] The batch size is preferably 8 to 32, which can be adjusted according to the size of the video memory.

[0280] For unlabeled data, weak augmentation is used for the teacher network and strong augmentation for the student network. Geometric augmentation maintains multimodal synchronization, color augmentation is performed only for the visible light mode, and thermal intensity perturbation is performed only for the infrared mode.

[0281] The optimal EMA coefficient for the teacher network is 0.999.

[0282] If there are very few new category labels, episode sampling can be used, that is, random sampling is performed for each training episode. Each category, each class One support sample and several query samples are used to enhance the prototype learning effect.

[0283] (III) Disclosure of Parameter Selection

[0284] In the formula for modal reliability coefficient ~ The preferred value is described in step S2 above;

[0285] In the formula for scene complexity index ~ It can be adjusted according to the application scenario;

[0286] In the formula of hardware resource indicators ~ It can be adjusted according to the bottleneck type of the deployment platform;

[0287] Prototype update coefficient The preferred value is 0.90 to 0.98;

[0288] Teacher EMA coefficient The preferred value is 0.99 to 0.9995;

[0289] The threshold for filtering false labels can be adjusted based on the validation set, with a preferred strong reliability threshold of 0.80 and a medium reliability threshold of 0.65.

[0290] When the number of samples in the new category is extremely small, it is preferable to appropriately increase... and This is to enhance prototype constraints and consistency constraints.

[0291] (iv) Disclosure of Dataset Use

[0292] If the dataset is built by the enterprise itself, it is preferable to ensure that the basic class annotation set is sufficient, the unannotated set is large in scale, and the new class annotation is very few.

[0293] If using a combination of public datasets, you can first pre-train on general multimodal public datasets, and then fine-tune on self-built datasets for the target scenario;

[0294] The definition of the annotation box should be unified to the reference modal coordinate system, and then mapped to other modes;

[0295] The dataset should be divided into at least a training set, a validation set, and a test set. The validation set is used to select the threshold and loss weights, and the test set is used only for the final evaluation.

[0296] For small sample settings, it is preferable to keep the sampled instances in the validation set different from those in the training set in order to avoid overfitting.

[0297] V. Specific Application Examples

[0298] (I) Application Example 1: Experiment on Defect Detection of Dual-Mode Scarce Samples under Low Illumination and Thermal Interference Conditions

[0299] This embodiment is used to verify the detection performance of the present invention under complex conditions such as scarce labeled samples, low illumination, partial occlusion, and thermal background interference. The detection targets are abnormal targets on the surface and near-surface of composite board materials, and dual-modal input of visible light and infrared images is used. Its technical approach is similar to... Figure 1 The system structure shown Figure 2 The method flow shown Figure 3 Modal reliability calculation shown Figure 4 The fusion path selection shown Figure 5 The pseudo-label filtering and prototype update shown Figure 6 The teacher-student collaborative training process shown corresponds to this.

[0300] The experimental platform includes a 5-megapixel industrial camera and a camera with a resolution of [missing information]. The system includes a long-wave infrared camera, a synchronous trigger controller, an industrial lighting module, an adjustable-speed conveyor platform, and a GPU training server. The industrial camera and infrared camera are fixedly mounted above the conveyor platform, with a field of view coverage of 650mm and a acquisition distance of 600mm. Visible and infrared images are synchronized using a unified timestamp, spatially registered using a calibration board, and images of different modalities are uniformly scaled to [a specific scale]. Enter the dimensions.

[0301] This embodiment constructs a dual-modal defect detection dataset, containing a total of 23,420 pairs of visible light-infrared samples. The dataset includes eight basic categories: scratches, indentations, stains, edge defects, holes, blistering, foreign objects, and hot spots, totaling 4,800 labeled samples. Four new categories are included: fine cracks, delamination, poor overlap, and ablation spots, with only five labeled samples retained for each category, totaling 20 labeled samples to simulate scarce sample scenarios. There are also 15,000 unlabeled samples. The validation set contains 1,600 pairs, and the test set contains 2,000 pairs. The test set is further divided into five sub-scenarios based on operating conditions: normal lighting, low light, strong reflection, partial occlusion, and thermal background interference, to examine robustness in each sub-scenarios.

[0302] In this embodiment, both the student detection network and the teacher detection network use the same lightweight multimodal detection backbone network, with an input size of [insert size here]. The batch size is 16, the optimizer is AdamW, and the initial learning rate is... The weight decay is 0.01. Warm-up training is performed for 20 epochs using only labeled samples; joint training is then conducted for 100 epochs using both labeled and unlabeled samples. Teacher network parameters are updated using an exponential moving average with a moving average coefficient of 0.999. The routing threshold is set to the scene complexity threshold. Hardware resource threshold Modal confidence threshold The pseudo-label classification thresholds are set as follows: 0.80 for strong reliability and 0.65 for medium reliability.

[0303] This embodiment uses the following indicators to evaluate the detection effect: accuracy. Calculate using the following formula:

[0304] ;

[0305] in, This indicates the number of true positives. This indicates the number of false positives.

[0306] Recall rate Calculate using the following formula:

[0307] ;

[0308] in, This indicates the number of false negatives.

[0309] Comprehensive indicators Calculate using the following formula:

[0310] ;

[0311] in, Indicates accuracy. This indicates the recall rate.

[0312] Where mAP@0.5 represents the average precision of each category when the intersection-union ratio (IU) threshold is 0.5; mAP@0.5:0.95 represents the average precision when the IU threshold ranges from 0.5 to 0.95 with a step size of 0.05. For the pseudo-label quality of unlabeled samples, this embodiment uses the pseudo-label precision. The evaluation is calculated as follows:

[0313] ;

[0314] in, This indicates the number of pseudo-labels that were screened and confirmed as correct through manual sampling or comparison with the retained true values. This represents the total number of pseudo-labels selected for training.

[0315] To verify the technical effect of the present invention, the following comparison method is set in this embodiment:

[0316] Comparative Example 1 is a YOLOv8n detection method that uses only the visible light single mode;

[0317] Comparative Example 2 is the YOLOv8n detection method using only infrared single-mode;

[0318] Comparative Example 3 is a fixed early fusion dual-modal detection method;

[0319] Comparative Example 4 is a fixed late-fusion dual-modal detection method;

[0320] Comparative Example 5 uses a teacher-student semi-supervised + fixed-center integrated approach;

[0321] The implementation method is the complete method of modal credibility closed-loop control + dynamic fusion path selection + prototype memory + hierarchical pseudo-label screening described in this invention.

[0322] Table 1 presents the overall performance comparison results of different methods on the test set.

[0323] Table 1. Overall performance comparison of different methods

[0324]

[0325] As shown in Table 1, compared with Comparative Example 5, the mAP@0.5 of the present invention improved by 2.7 percentage points, the mAP@0.5:0.95 improved by 4.1 percentage points, the recall improved by 3.1 percentage points, the F1 score improved by 2.8 percentage points, and the mAP@0.5 for new categories improved by 6.6 percentage points. This indicates that the present invention not only improves the overall detection accuracy but also has better recognition capabilities for new category targets with only a small number of labeled samples. Combined with... Figure 7 As can be seen, the present invention achieves a better accuracy-speed balance while maintaining high real-time performance, indicating that the modal reliability closed-loop control and dynamic path selection in the present invention do not cause unacceptable inference overhead.

[0326] To further illustrate the recognition gain of the new categories by the present invention, Table 2 lists the detection results of the four new categories under different methods.

[0327] Table 2 Comparison of Detection Results for New Categories

[0328]

[0329] As shown in Table 2, the improvement of this invention is particularly significant in the categories of fine cracks and delamination, where there are fewer samples and more obvious morphological differences. Specifically, the improvement for the fine crack category is 9.1 percentage points compared to Comparative Example 5, and for the delamination category, it is 7.3 percentage points. This indicates that the category prototype memory and hierarchical pseudo-label screening set in this invention can effectively suppress the problems of category drift and erroneous supervision diffusion under conditions of few samples, thereby improving the stable representation ability of new categories. Combined with... Figure 5 and Figure 9 This is understandable, as it is because introducing only highly reliable or moderately reliable pseudo-labels into training significantly reduces the interference of erroneous pseudo-labels on the prototype update direction.

[0330] To verify the robustness of the present invention under complex working conditions, Table 3 lists the recall results under different sub-scenarios.

[0331] Table 3 Comparison of recall rates under different complex operating conditions

[0332]

[0333] As shown in Table 3, the recall rate of this invention is significantly improved in complex scenarios such as low light, partial occlusion, and thermal background interference. Particularly in low light scenarios, this invention improves recall by 6.7 percentage points compared to Comparative Example 5; and in thermal background interference scenarios, it improves recall by 7.5 percentage points. This indicates that this invention does not simply superimpose visible light and infrared information, but rather... Figure 3 The modal confidence coefficient calculation shown dynamically suppresses the negative impact of low-confidence modes on the final detection; simultaneously, based on Figure 4 The fusion path selection mechanism shown automatically calls the more suitable mid- or late-fusion path in complex scenarios, thereby significantly enhancing adaptability to complex environments.

[0334] In this embodiment, some samples were also subjected to visual analysis. Figure 11 It is recommended to select the following four sets of typical images: The first set consists of fine crack samples with low illumination and weakened texture, which are missed by RGB single-mode detection, while the present invention can achieve correct identification by relying on infrared supplementary information and prototype constraints; the second set consists of hot spot samples with strong thermal background interference, which produce false detections by IR single-mode detection, while the present invention suppresses erroneous thermal responses through modal confidence reduction and late fusion; the third set consists of poorly overlapping samples with partial occlusion, where fixed early fusion methods introduce occlusion noise into low-level features, causing detection box offset, while the present invention maintains correct positioning; the fourth set consists of delamination samples of new categories, where fixed fusion methods have low confidence, while the present invention achieves stable detection through prototype memory and hierarchical pseudo-label mechanism. These visualization results can further intuitively demonstrate the technical effectiveness of the present invention.

[0335] (II) Application Example 2: Online Detection and Verification Experiment under Edge Deployment Conditions

[0336] This embodiment is used to verify the real-time performance and resource adaptability of the present invention on edge devices. The model trained in Application Example 1 is deployed on a Jetson Orin NX16GB edge computing terminal, using a 20W power consumption mode, and running in an environment of Ubuntu 22.04, CUDA 12.2, and TensorRT inference engine. The test sample consists of 1000 frames of bimodal images continuously collected from the production line, of which simple scenes account for 64%, medium-complex scenes account for 24%, and complex scenes account for 12%.

[0337] To evaluate the effectiveness of online deployment, this embodiment uses average inference latency. As a performance indicator, its calculation formula is as follows:

[0338] ;

[0339] in, Indicates the total number of test frames. Indicates the first The total inference time from input to output of a frame.

[0340] The comparison methods include:

[0341] Comparative Example 6 uses a fixed high-precision path, always employing the fusion-enhanced detection path;

[0342] Comparative Example 7 uses a fixed low-cost path, always enabling the low-cost early integration path;

[0343] The embodiments of the present invention are dynamic path selection methods based on scene complexity and hardware resource status.

[0344] Table 4 lists the deployment results at the edge.

[0345] Table 4. Deployment performance comparison on edge devices

[0346]

[0347] As shown in Table 4, compared to a fixed high-precision path, the embodiments of the present invention reduce average latency by 7.6ms, maximum latency by 11.8ms, and average power consumption by 3.3W with only a 0.5 percentage point decrease in mAP@0.5. Compared to a fixed low-cost path, the present invention improves mAP@0.5 by 2.4 percentage points, indicating that the present invention can achieve a better latency-power-precision balance on edge devices. Combined with... Figure 10 It can be further seen that the present invention does not always run the high-energy-consuming detection path, but rather according to... Figure 4 The path switching logic shown prioritizes low-cost paths for simple scenarios and high-precision paths for complex scenarios, thereby achieving the effect of allocating computing power according to the scenario.

[0348] In this sample of 1000 frames, the present invention automatically selected 612 low-cost paths, 278 mid-range fusion paths, and 110 late-range fusion robust paths. This demonstrates that the dynamic routing of the present invention exhibits a clear scene selection pattern, rather than random switching, which is consistent with... Figure 4 , Figure 6 The system logic shown corresponds to the model, and further proves the effectiveness of the joint decision-making based on scenario complexity index, hardware resource index, and modal credibility in this invention.

[0349] (III) Application Example 3: Ablation Experiment

[0350] To further illustrate the contribution of the key technical features of this invention to the final technical effect, ablation experiments were conducted under the same dataset, training parameters, and testing conditions as in Application Example 1. The following model version was set:

[0351] Group A is a basic bimodal semi-supervised detection model that does not introduce modal reliability closed-loop control, dynamic routing, category prototype memory, or hierarchical pseudo-label filtering.

[0352] Group B adds modal reliability calculation to the basic calculation of Group A;

[0353] Group C adds dynamic routing to the basic structure of Group B;

[0354] Group D adds category prototype memory to the foundation of Group C;

[0355] Group E adds hierarchical pseudo-label filtering to the group D;

[0356] Group F represents the complete method of this invention, further incorporating degradation detection sub-paths and resource constraint switching control.

[0357] To evaluate the stability of prototype updates, this embodiment introduces prototype drift distance. The calculation formula is as follows:

[0358] ;

[0359] in, Indicates the total number of categories. Indicates the first The first round of updates class prototype vector, Indicates the number of the previous round class prototype vector, This represents the cosine similarity. The smaller the value, the more stable the prototype updates.

[0360] Table 5 shows the results of the ablation experiment.

[0361] Table 5 Ablation Experiment Results

[0362]

[0363] As shown in Table 5, the modal reliability mechanism is the primary key factor in improving the accuracy of pseudo-labels and the detection capability in complex scenes. Dynamic routing, while improving accuracy, also reduces the average latency from 27.8ms to 25.9ms, indicating its direct value on the deployment side. Category prototype memory significantly improves the detection performance of new categories and reduces the prototype drift distance from 0.110 to 0.073. Hierarchical pseudo-label screening further reduces error supervision propagation, improving pseudo-label accuracy to 88.5%. Based on this, the complete method further optimizes latency performance through degraded detection sub-paths and resource constraint control. Figure 9 and Figure 10 This clearly demonstrates that the components of this invention are not simply parallel, but rather coupled together and work together to achieve the final technical effect.

[0364] To further illustrate the trends during the training process, Table 6 presents the pseudo-label accuracy and prototype drift distance data for the complete method at different epochs during the joint training phase, which can be directly used to plot data. Figure 9 .

[0365] Table 6. Changes in pseudo-label accuracy and prototype drift distance during training.

[0366]

[0367] As shown in Table 6, the accuracy of the pseudo-labels continuously improves while the prototype drift distance continuously decreases as training progresses. This indicates that the closed-loop mechanism of this invention, consisting of modality reliability, hierarchical pseudo-labels, and prototype memory updates, can gradually improve the quality of pseudo-labels and stabilize the category prototype. This is consistent with... Figure 5 The prototype update path shown Figure 6 The teacher-student collaborative training process shown is consistent with the actual process, which further proves that the technical effect of the present invention is reflected not only in the final accuracy, but also in the stability of the training process.

[0368] The foregoing description of embodiments of the present invention, through which those skilled in the art are able to implement or use the present invention, will be readily apparent to those skilled in the art. Various modifications to these embodiments will be readily apparent to those skilled in the art. The general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novelty disclosed herein.

[0369] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0370] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0371] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0372] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0373] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0374] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0375] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

Claims

1. A lightweight visual detection method for multimodal scarce samples based on modal reliability closed-loop control, characterized in that, Includes the following steps: S1. Obtain raw data of at least two modalities of the scene to be detected, and complete cross-modal spatiotemporal registration using a unified timestamp and calibration parameters to construct a labeled set containing a small number of labeled samples, an unlabeled set containing unlabeled samples, and an online sample set to be detected. S2. Input the original data of the at least two modalities into the shared lightweight backbone network and the modality-specific feature encoding branch respectively, extract the multi-scale features of each modality, and calculate the modality confidence coefficient of each modality based on the alignment residual, noise index and effective information index between the multi-scale features of each modality and the reference alignment features. S3. Based on the modality confidence coefficient of each modality and the scene complexity index of the current sample, select at least one fusion path from the preset early fusion path, mid fusion path and late fusion path to perform hierarchical feature fusion. When the modality confidence coefficient of any modality is lower than the first threshold, the contribution of that modality to the fusion result is suppressed or the detection path is switched to a degraded one to obtain the fusion feature representation. S4. Construct a category prototype memory based on the labeled set, and use a teacher detection network to generate candidate boxes and pseudo-labels for the unlabeled set; Based on the pseudo-label confidence score, cross-modal consistency score, and modal confidence coefficient of each modality, the pseudo-labels are jointly screened, and only pseudo-labels that meet the reliability conditions are used for classification loss and localization loss calculation. The category prototype memory is updated using samples that meet the reliability conditions. S5. Input the fused feature representation, the category prototype memory, and the filtered pseudo-labels into the student detection network for training, and iteratively update the teacher detection network parameters based on the student detection network parameters. During the inference phase, preset detection sub-paths are dynamically activated based on scene complexity and hardware resource metrics, and the target category, target location, and detection confidence are output.

2. The method according to claim 1, characterized in that, In step S1, the at least two modal raw data include any two or more of visible light images, infrared images, depth images, lidar point clouds, and millimeter-wave radar echoes; the cross-modal spatiotemporal registration includes: completing frame-level alignment according to a unified timestamp, and mapping each modal data to a unified spatial coordinate system according to the intrinsic parameter matrix, extrinsic parameter matrix, projection matrix, or point cloud to image mapping relationship.

3. The method according to claim 1, characterized in that, In step S2, the shared lightweight backbone network is a depthwise separable convolutional network, a Ghost network, a lightweight Transformer network, or a combination thereof; the modality-specific feature encoding branches are used to extract texture features, thermal radiation features, geometric features, or reflection intensity features, respectively. The noise index is used to characterize the noise intensity, missing degree, or distortion degree of the current modality; the effective information index is used to characterize the distinguishable texture, edge, geometric structure, or target response intensity of the current modality in the current scene.

4. The method according to claim 1, characterized in that, In step S2, the modal reliability coefficient Calculate using the following formula: ； In the formula, Indicates the first Modal reliability coefficients for each mode; Indicates the total number of modes; Indicates the first Alignment residuals for each mode; Indicates the first Noise index for each mode; Indicates the first Effective information indicators for each modality; Indicates the first Data missing rate for each modality; , , and These are weighting coefficients, and all are real numbers greater than 0.

5. The method according to claim 1, characterized in that, In step S3, the scene complexity index and the aforementioned hardware resource indicators Calculate according to the following formulas: ； ； In the formula, Indicates the complexity of the scenario; Indicates the density of candidate targets; Indicates the target overlap rate; Indicates a blurred or obscured indicator; Indicators representing fluctuations in illumination or motion disturbances; , , , The weighting factor is the scene complexity. Indicates hardware resource metrics; Indicates the percentage of remaining memory; Indicates the proportion of available computing power; Indicates the remaining power consumption budget percentage; , , The weighting factor is the hardware resource weighting factor. when Activate the low-cost detection sub-path at the time; when and Time-activated precision enhancement detection subpath; when any mode of The degradation detection sub-path or single-modal backup path is activated at the appropriate time; among which... , , This is a preset threshold.

6. The method according to claim 1, characterized in that, In step S4, the category prototype memory adopts an exponential sliding update method, and the category... prototype vector Update as follows: ； In the formula, Indicates the first The updated categories Prototype vector; Indicates the first The updated categories Prototype vector; This represents the prototype update smoothness coefficient, and ; This indicates that the product passed the reliability screening and belongs to the relevant category. The number of samples; Indicates the first Each belongs to category The fused feature vector of the samples.

7. The method according to claim 1 or 6, characterized in that, In step S4, the reliability score of the pseudo-tag is... Calculate using the following formula: ； In the formula, Indicates the first Reliability score of each pseudo-label; Indicates the first Confidence level of pseudo-label detection; Indicates the first Cross-modal consistency score of pseudo-labels; Indicates the first Modal reliability coefficients for each mode; Indicates the first The pseudo-label is in the first Modal support scores on each modality; Indicates the first The distance penalty term between the fused features of the sample corresponding to each pseudo-label and its class prototype; , , , These are weighting coefficients, and all are real numbers greater than 0; when and At that time, the first A pseudo-label is used simultaneously for calculating both classification loss and localization loss; when the above conditions are not met, the pseudo-label is written to the uncertain sample buffer and does not participate in the current round of training; where... and This is a preset threshold.

8. A lightweight multimodal scarce sample visual inspection system based on modal reliability closed-loop control, characterized in that, The system is used to implement the method according to any one of claims 1-7, comprising: The data acquisition and registration module is used to acquire raw data from at least two modalities and complete spatiotemporal registration. The feature extraction module is used to extract multi-scale features of each modality using a shared lightweight backbone network and modality-specific feature encoding branches; The modal credibility calculation module is used to calculate the modal credibility coefficient based on alignment residuals, noise index, effective information index, and data missing rate. The fusion path selection module is used to select the fusion path and detection sub-path based on the modality confidence coefficient, scene complexity index, and hardware resource index. The pseudo-label filtering and prototype update module is used to generate pseudo-labels, calculate pseudo-label reliability scores, and update the category prototype memory. The teacher-student collaborative training module is used to train the student detection network and update the teacher detection network based on the fused feature representation, the category prototype memory, and the filtered pseudo-labels. The results output module is used to output the target category, target location, and detection confidence.

9. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images