Electric energy meter single board defect detection method and system based on multi-modal data fusion

Abstract

The application discloses a kind of electric energy meter single board defect detection method and system based on multimodal data fusion, it is related to electric energy meter intelligent quality detection technical field, including synchronous acquisition electric energy meter single board image and carry out fusion pretreatment, obtain fusion image data;Defect detection model is constructed, and defect detection model is trained;Real-time inference is carried out to fusion image data using trained defect detection model, generates defect detection result and evaluates defect detection result confidence degree.The electric energy meter single board defect detection method based on multimodal data fusion provided in the application realizes the synchronous acquisition of electric energy meter single board multimodal information and feature enhancement, provides high-quality input data for subsequent detection model, realizes the deep feature modeling of multiple defects and the optimization of recognition ability, realizes the whole process inference mechanism of defect space positioning, type classification and reliability grading, finally reaches defect annotation accurate, convenient automatic sorting and quality closed-loop control.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent quality detection of electric energy meters, and in particular to an electric energy meter panel defect detection method and system based on multimodal data fusion. Background Art

[0002] With the continuous advancement of the State Grid's digitalization and intelligent power distribution and utilization systems, smart energy meters have become a key device for energy measurement, data collection, and terminal interaction. As a core component, the structure and welding quality of the meter's mainboard directly impact the metering accuracy and reliability of the entire device. In recent years, with the integration and miniaturization of electronic components, the layout of meter mainboards has become increasingly compact, and component density has significantly increased. This places higher demands on the integrity and consistency of solder joints during the manufacturing process. Currently, most energy meter manufacturers still rely on manual visual inspection or traditional optical image acquisition systems to control solder joint quality. This method can meet basic screening requirements in low-defect scenarios, but its detection efficiency and accuracy are gradually becoming insufficient in the context of mass production and multi-model parallel processing. In particular, for defects such as cold solder joints, residual solder balls, and bridge shorts, which are small in size, hidden in location, or have complex causes, relying solely on single-channel image signal sources (such as RGB visible light images) is unable to fully characterize all defects. Furthermore, some solder joints may be located in areas covered by silkscreen ink or obscured by the back of the circuit board. Traditional cameras have inherent limitations in imaging angle and penetration, which can easily lead to loss or misidentification of key defect information, resulting in high false detection or missed detection rates. To address this issue, researchers have gradually introduced heterogeneous data sources such as infrared imaging, multi-band image acquisition, and depth maps, using multimodal image fusion as an effective supplementary means of defect detection. However, the real-time, robustness, and deep feature extraction capabilities of the fusion algorithm are still limited, making it difficult to meet the stringent requirements of industrial sites for intelligent, real-time, and low-error detection systems.

[0003] Extensive research has been conducted in defect detection algorithms, image fusion methods, and recognition model architectures. For example, at the image processing level, some studies have attempted to employ image fusion techniques based on edge enhancement and regional feature contrast enhancement to enhance the saliency of solder joints in images. At the model training level, deep learning architectures such as convolutional neural networks (CNNs) and region proposal networks (RPNs) have been introduced for the automatic labeling and classification of defect targets. While these solutions have achieved some success in small sample sizes or experimental scenarios, they generally suffer from the following technical limitations: Traditional fusion strategies, most of which rely on static weighting or rule-driven approaches, lack the ability to adapt to local image features and struggle to maintain stable performance under dynamic backgrounds or complex interference conditions. Limited training data sources and sample imbalance (normal solder joints far outnumber cold or abnormal solder joints) often lead to overfitting or class bias in network models, ultimately affecting model generalization and detection accuracy. Existing defect detection models mostly rely on GPU platforms for time-consuming training and inference, failing to effectively address the real-time inference requirements of edge computing environments. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problems solved by the present invention are: the existing defect detection methods based on a single image modality have the problems of insufficient recognition accuracy, high missed detection rate, difficulty in coping with complex background interference and limited ability to recognize hidden defects, as well as how to achieve accurate recognition and confidence quantification evaluation of electric energy meter sheet defects based on multimodal image fusion.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: an electric energy meter panel defect detection method based on multimodal data fusion, comprising synchronously collecting electric energy meter panel images and performing fusion preprocessing to obtain fused image data; constructing a defect detection model and training the defect detection model; using the trained defect detection model to perform real-time inference on the fused image data, generating defect detection results and evaluating the confidence of the defect detection results; the fusion preprocessing comprises obtaining a visible light image and a penetration image of the electric energy meter panel, constructing a weighted fusion rule using the local contrast of the image, performing weighted fusion on the RGB image and the NIR image, and obtaining fused image data; generating defect detection results and evaluating the confidence of the defect detection results comprises inputting the fused image data into the trained defect detection model for feature extraction and target recognition, obtaining a structured defect detection result including the defect location, type and classification confidence, calculating the confidence of the defect detection result, and comparing it with the confidence threshold to determine the reliability and processing path of the defect detection result.

[0007] As a preferred solution of the electric energy meter board defect detection method based on multimodal data fusion described in the present invention, the synchronous acquisition of the electric energy meter board image and fusion preprocessing includes capturing the visible light image of the electric energy meter board through an RGB camera to obtain an RGB image, and capturing hidden defects through the ink on the electric energy meter circuit board through an NIR camera to obtain an NIR image, and preprocessing the RGB image and the NIR image.

[0008] As a preferred solution of the electric energy meter sheet defect detection method based on multimodal data fusion described in the present invention, the obtaining of fused image data includes calculating a fusion weight coefficient based on the local contrast of the image, performing weighted fusion processing on the preprocessed RGB image and NIR image, and obtaining fused image data.

[0009] As a preferred solution of the electric energy meter panel defect detection method based on multimodal data fusion described in the present invention, the training of the defect detection model includes constructing an image training set containing defect samples and normal samples, constructing a lightweight detection backbone network, training the defect detection model through the image training set, configuring a preset input size and optimization strategy, and optimizing the defect detection model parameters based on a preset round of training iterations.

[0010] As a preferred solution of the electric energy meter panel defect detection method based on multimodal data fusion described in the present invention, the real-time reasoning of the fused image data includes inputting the fused image data into the trained defect detection model for frame-by-frame analysis, combining the multi-scale feature extraction structure in the defect detection model, generating defect detection results and parsing the defect detection results.

[0011] As a preferred solution of the electric energy meter panel defect detection method based on multimodal data fusion described in the present invention, the parsing of the defect detection results includes parsing the defect detection results output by the real-time reasoning of the defect detection model to obtain the defect location and defect type.

[0012] As a preferred solution of the electric energy meter single board defect detection method based on multimodal data fusion described in the present invention, the evaluation of the confidence of the defect detection result includes constructing a confidence evaluation function based on the sample distribution characteristics, classification difficulty and boundary characteristics in the defect detection model training process, and comprehensively evaluating the defect location and defect type in combination with the spatial positioning matching degree and feature response intensity of the potential defects, calculating the confidence of the defect detection result, and comparing the confidence of the defect detection result with the confidence threshold. If the confidence of the defect detection result is higher than the confidence threshold, the defect detection result is determined to be valid defect information, and the single board is calibrated as an abnormal single board.

[0013] If the confidence level of the defect detection result is lower than the confidence threshold, the defect detection result is judged as invalid defect information, and a re-inspection instruction is triggered.

[0014] Another object of the present invention is to provide an electric energy meter panel defect detection system based on multimodal data fusion. By integrating the image processing module, the neural network training module, and the defect detection and confidence assessment module, the system can solve the technical bottlenecks of current image detection technology in terms of weak multi-source data fusion capability, unquantifiable model reasoning credibility, and low on-site deployment response efficiency.

[0015] As a preferred solution of the electric energy meter panel defect detection system based on multimodal data fusion described in the present invention, it includes: a fusion image processing module, a neural network training module, and a defect detection and confidence assessment module; the fusion image processing module includes an image acquisition unit and a weighted fusion unit, the image acquisition unit is used to synchronously acquire multimodal image data of the electric energy meter panel through an RGB camera and an NIR camera, and the weighted fusion unit is used to perform fusion preprocessing on the acquired image data; the neural network training module is used to construct a fusion image training set containing multiple categories of defect samples and normal samples, perform multiple rounds of training and parameter updates based on the target detection loss function, and dynamically adjust the model structure and training strategy based on sample features and training feedback results; the defect detection and confidence assessment module includes a defect reasoning unit and a confidence assessment unit, the defect reasoning unit is used to input the fusion image data into the trained defect detection model and output the defect detection result, and the confidence assessment unit is used to calculate the confidence index of each defect target in the model reasoning result, and compare it with the preset threshold, output a high-confidence defect or trigger a re-inspection mechanism.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a method for detecting defects in an electric energy meter panel based on multimodal data fusion.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for detecting defects in an electric energy meter panel based on multimodal data fusion.

[0018] Beneficial effects of the present invention: The electric energy meter board defect detection method based on multimodal data fusion provided by the present invention obtains fused image data by synchronously collecting electric energy meter board images and performing fusion preprocessing, thereby realizing the synchronous acquisition and feature enhancement of multimodal information of the electric energy meter board, providing high-quality input data for subsequent detection models, and constructing a defect detection model and training the defect detection model to realize deep-level feature modeling and recognition capability optimization for multiple types of defects, establish a lightweight and efficiently deployable detection network, and realize a full-process reasoning mechanism for defect spatial positioning, type classification and credibility grading. Its function is to convert the model reasoning results into executable decision outputs, and ultimately achieve accurate defect labeling, credible judgment results, and facilitate automatic sorting and quality closed-loop control. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is an overall flow chart of the electric energy meter panel defect detection method based on multimodal data fusion provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0021] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0022] Example 1, reference Figure 1 , as one embodiment of the present invention, provides a method for detecting defects in an electric energy meter panel based on multimodal data fusion, comprising:

[0023] S1: Synchronously collect the electric energy meter panel image and perform fusion preprocessing to obtain fused image data.

[0024] Furthermore, synchronously collecting the image of the electric energy meter board and performing fusion preprocessing includes capturing the visible light image of the electric energy meter board through an RGB camera to obtain an RGB image, and using a NIR camera to penetrate the ink on the electric energy meter circuit board to capture hidden defects to obtain an NIR image, and preprocessing the RGB image and NIR image.

[0025] It should be noted that RGB cameras and NIR cameras with industrial-grade imaging accuracy are selected as image acquisition devices, which are used to obtain visible light images and near-infrared penetration images of the electric energy meter board respectively, to ensure multi-source capture of the surface texture and internal morphology information of the solder joint area. Position calibration and time synchronization configuration are performed during camera installation to ensure the consistency of the RGB image and NIR image channels in the shooting time and spatial position. In the image acquisition stage, the images collected by the RGB camera are mainly used to reflect the structure, text markings, solder joint edges and other texture information on the surface of the circuit board; the NIR camera penetrates part of the ink coating and shielded areas to supplement the morphological characteristics of defects such as hidden solder joints and fine tin bead residues. The environmental noise interference caused by the acquisition is removed by Gaussian filtering, and the standard deviation of the Gaussian filter is set. , grayscale normalization is performed on RGB images and NIR images, and pixel values ​​are linearly mapped to the standard grayscale range of 0~255.

[0026] It should also be noted that by using RGB cameras and NIR cameras to collaboratively collect images of the power meter board, and performing position calibration and time synchronization configuration during the installation phase, the simultaneous acquisition of multimodal and multi-dimensional features of the circuit board welding area is achieved, and a cross-modal image input structure with high alignment accuracy is constructed, which improves the quality of defect recognition front-end data and enhances the detection system's sensitivity and recognition range for hidden and micro defects.

[0027] Furthermore, obtaining the fused image data includes calculating a fusion weight coefficient according to a local contrast of the image, performing weighted fusion processing on the pre-processed RGB image and the NIR image, and obtaining the fused image data.

[0028] It should be noted that for the preprocessed RGB image and NIR image, the local contrast is calculated at the pixel level and expressed as:

[0029] ;

[0030] in, represents the local contrast, represents the preprocessed visible light image data, Represents the preprocessed near-infrared image data.

[0031] The fusion weight coefficient is calculated by local contrast and is expressed as:

[0032] ;

[0033] in, Indicates the pixel position The fusion weight coefficient at Represents pixel points The local contrast, Represents the maximum local contrast at all pixel positions in the fused image data, Represents the local contrast minimum at all pixel locations in the fused image data.

[0034] The weighted fusion of RGB image and NIR image is expressed as:

[0035] ;

[0036] in, Indicates the fused image at pixel position The final grayscale value, Represents an RGB image at pixel location The gray value of Indicates the NIR image at pixel position Gray value.

[0037] It should also be noted that by calculating the local contrast, the significant differences in different modal images at each pixel point are quantified, which provides a criterion for the subsequent weight allocation of fusion. By calculating the fusion weight coefficient, the boundary tearing or fusion artifacts of the fused image are prevented, the imaging consistency is improved, and the RGB image and NIR image are weightedly fused to achieve on-demand fusion, that is, the RGB image tends to be retained in the feature-salient area (high contrast), and the NIR image tends to be retained in the area with weak texture details or ink occlusion, to ensure the integrity of the information in the key areas.

[0038] S2: Build a defect detection model and train the defect detection model.

[0039] Furthermore, training the defect detection model includes constructing an image training set containing defect samples and normal samples, building a lightweight detection backbone network, training the defect detection model through the image training set, configuring the preset input size and optimization strategy, and optimizing the defect detection model parameters based on the preset rounds of training iterations.

[0040] It should be noted that the YOLOv7-Tiny lightweight network structure is preferably used as the basic network model of the defect detection model, the size of the input image is set to 640×640, and the Adam optimizer is used to update and optimize the parameters of the defect detection model. The batch_size in the training process is set to 16, and the total number of training rounds is 5000 rounds to ensure that the defect detection model can still effectively learn complex features under limited resources. An image training set containing defect samples and normal samples is constructed. The defect samples cover common defect types such as cold solder joints, wrong solder joints, pad offsets, and solder residues. Normal samples are used to construct normal background feature distributions. CIoU-Loss and Focal-Loss are set as loss functions to overcome the training bias of the defect detection model caused by sample imbalance. CIoU-Loss is used to guide the defect detection model to learn the target position and shape differences more accurately during the training process, effectively optimizing the overlap and geometric features between the prediction box and the actual defect area. Focal-Loss is used to introduce higher loss weights for a small number of difficult-to-detect samples such as cold solder joints, guiding the defect detection model to The model improves the learning ability of small samples of cold solder joints. According to the prediction error of different types of samples (cold solder joints, normal solder joints, etc.) in each round of training, the defect detection model optimization strategy is configured, and the weight coefficients of CIoU-Loss and Focal-Loss are dynamically adjusted. When the detection error of the defect detection model on the cold solder joint sample is large (preferably the IoU error is greater than 0.35), the weight ratio of Focal-Loss is increased. It is preferred to set the Focal-Loss weight to 0.6 and the CIoU-Loss weight to 0.4 to increase the model's learning of difficult-to-detect samples; when the defect detection model has large deviations in position and shape (preferably the average IoU is less than 0.7), it is preferred to dynamically increase the weight of CIoU-Loss to 0.7 and adjust Focal-Loss to 0.3 to more effectively guide the improvement of the prediction box regression accuracy.

[0041] It should also be noted that by setting the IoU error threshold, the problems of difficult classification of cold solder joints and inaccurate positioning of prediction frames in the current training round of the defect detection model can be dynamically identified, and the weight of the loss function can be adjusted to ensure the reasonable allocation of learning resources of the defect detection model. By establishing a dynamic adjustment mechanism, better detection accuracy and recall rate can be achieved under the same number of training rounds, which is suitable for industrial image detection scenarios with uneven sample distribution and large feature differences.

[0042] S3: Use the trained defect detection model to perform real-time inference on the fused image data, generate defect detection results, and evaluate the confidence of the defect detection results.

[0043] Furthermore, real-time reasoning of the fused image data includes inputting the fused image data into the trained defect detection model for frame-by-frame analysis, combining the multi-scale feature extraction structure in the defect detection model, generating defect detection results, and parsing the defect detection results.

[0044] It should be noted that the defect detection model is expressed as:

[0045] ;

[0046] in, represents the defect detection result set, Indicates the three output scales under multi-scale feature extraction, that is, the output resolution is 1 / 8, 1 / 16, and 1 / 32 of the input. Represents the union of the output results at three output scales, Indicates that the The features at different scales are fused. represents the lightweight detection backbone network, Indicates in The network parameter set obtained by training at different scales.

[0047] The fused image data is input into the lightweight detection backbone network to extract basic features with semantic information. The lightweight detection backbone network outputs feature maps at different resolutions according to three scales. Perform feature fusion to improve the detection ability of multi-scale targets (especially small defects).

[0048] It should also be noted that by constructing a scale set and extracting feature maps at each scale and then fusing them, the defect detection model takes into account both global perception and local details, significantly improving the ability to identify defect targets of different sizes, especially smaller solder joint defects and subtle cold solder joints. YOLOv7-Tiny is used as the backbone network to achieve high detection accuracy while reducing the model calculation complexity. It is suitable for deployment in edge devices or online detection systems to improve the system's real-time performance and resource adaptability.

[0049] Furthermore, parsing the defect detection results includes parsing the defect detection results output by the real-time reasoning of the defect detection model to obtain the defect location and defect type.

[0050] It should be noted that the defect detection results are analyzed and expressed as:

[0051] ;

[0052] in, Indicates the The position of the predicted location box of the defect, Indicates the The class labels of defects, Indicates the number of targets obtained by parsing, Indicates scale The output result of the above code is used to perform a joint decoding operation of target box extraction and category label decoding.

[0053] Through the scale-corresponding decoding operation, the bounding box coordinates and defect type label of each defect target are extracted, and the structured output of the defect detection result is obtained, which includes the location information and classification information of each defect.

[0054] It should also be noted that by constructing a structured decoding expression and introducing a joint parsing mechanism for defect location boxes and defect type labels, it is possible to accurately restore and extract information from multi-scale defect detection results. This parsing process not only preserves the spatial information of the multi-scale output but also effectively extracts the boundary coordinates and corresponding classification information of each defect target, enabling the detection results to be output in a structured manner.

[0055] Furthermore, evaluating the confidence of defect detection results includes constructing a confidence evaluation function based on the sample distribution characteristics, classification difficulty and boundary features in the defect detection model training process, and comprehensively evaluating the defect location and defect type in combination with the spatial positioning matching degree and feature response intensity of potential defects, calculating the confidence of the defect detection results, and comparing the confidence of the defect detection results with the confidence threshold. If the confidence of the defect detection result is higher than the confidence threshold, the defect detection result is judged as valid defect information, and the single board is calibrated as an abnormal single board.

[0056] If the confidence level of the defect detection result is lower than the confidence threshold, the defect detection result is judged as invalid defect information, and a re-inspection instruction is triggered.

[0057] It should be noted that the confidence evaluation function is constructed as follows:

[0058] ;

[0059] in, Indicates the The confidence level of each defect detection result, represents the classification difficulty item weight, represents the classification difficulty metric function, represents the bounding box quality term weight, represents the bounding box confidence function, represents the weight of the sample distribution sparse term, represents the sample distribution scoring function.

[0060] Based on each defect target output by the defect detection model, the predicted position box and category label are extracted, and a confidence scoring function is constructed based on the sample distribution characteristics, classification difficulty and boundary features. The confidence scoring function includes: a sparsity measurement based on the category distribution, which reflects the frequency of defect categories in the training set. The fewer categories, the more difficult the classification result and the lower the score. The bounding box quality item is used to measure the position accuracy and size rationality of the target detection box. Combined with the distribution of defect categories in the training samples, it is quantified whether the sample is highly representative or whether it is a rare sample. A weighted combination is performed to form a comprehensive confidence score for each defect target.

[0061] Comparing the confidence of the defect detection result with the confidence threshold is expressed as:

[0062] ;

[0063] in, Indicates the The result of the defect information determination is: Represents the confidence threshold.

[0064] The confidence level of each defect detection result is compared with the preset confidence threshold. If the confidence level of the defect detection result is greater than the confidence threshold, it is determined to be valid defect information; if the confidence level of the defect detection result is lower than the confidence threshold, it is determined to be invalid defect information and the re-inspection mechanism is triggered.

[0065] A preferred solution for the confidence threshold is ,The effects of different thresholds on defect detection accuracy and recall rate were tested on the training set and the validation set. It has a better balanced F1 score among multiple types of defects, combined with the frequency distribution of low-quality samples (such as incomplete bounding boxes and uncertain categories) in the training set. It can better distinguish between false positives and valid targets, and trigger the recheck mechanism for low-confidence samples to ensure safety redundancy. Strike a balance between ensuring detection efficiency and controlling false positives.

[0066] It should also be noted that by constructing a confidence evaluation function through a weighted combination of three factors: sample distribution characteristics, classification difficulty, and boundary characteristics, the reliability of defect identification results can be more comprehensively reflected, effectively alleviating misjudgments caused by skewed data distribution, ambiguous classification, or ambiguous boundaries, and achieving comprehensive quantification and dynamic control of the credibility of defect detection results, significantly improving the system's interpretability, reliability, and security capabilities.

[0067] Example 2 is an embodiment of the present invention, which provides an electric energy meter panel defect detection system based on multimodal data fusion, including a fusion image processing module, a neural network training module, and a defect detection and confidence assessment module.

[0068] Among them: the fusion image processing module includes an image acquisition unit and a weighted fusion unit. The image acquisition unit is used to synchronously acquire multimodal image data of the electric energy meter panel through the RGB camera and the NIR camera. The weighted fusion unit is used to perform fusion preprocessing on the acquired image data.

[0069] It should also be noted that the fusion image processing module collects multimodal image data from the RGB camera and the NIR camera through the image acquisition unit. The collected images are sent to the weighted fusion unit to perform fusion preprocessing on the different modal image data, generate unified fusion image data and pass it to the neural network training module.

[0070] The neural network training module is used to construct a fusion image training set containing multiple categories of defect samples and normal samples, perform multiple rounds of training and parameter updates based on the target detection loss function, and dynamically adjust the model structure and training strategy by combining sample features and training feedback results.

[0071] It should also be noted that the neural network training module builds a detection model based on the fused image training set and performs multiple rounds of training and parameter updates according to the target detection loss function.

[0072] The defect detection and confidence assessment module includes a defect reasoning unit and a confidence assessment unit. The defect reasoning unit is used to input the fused image data into the trained defect detection model and output the defect detection results. The confidence assessment unit is used to calculate the confidence index of each defect target in the model reasoning result, and compare it with the preset threshold, output high-confidence defects or trigger a re-inspection mechanism.

[0073] It should also be noted that the defect inference unit is used to call the trained defect detection model, perform target inference on the input fused image data, and output preliminary defect detection results. The confidence assessment unit receives the inference results, calculates the quantitative confidence index for each detected defect target, and compares it with the preset confidence threshold.

Claims

1. The electric energy meter panel defect detection method based on multimodal data fusion is characterized by: include: Synchronously collect the electric energy meter panel image and perform fusion preprocessing to obtain fused image data; Build a defect detection model and train the defect detection model; Use the trained defect detection model to perform real-time inference on the fused image data, generate defect detection results, and evaluate the confidence of the defect detection results; Fusion preprocessing includes obtaining the visible light image and penetration image of the electric energy meter panel, constructing a weighted fusion rule based on the local contrast of the image, and performing weighted fusion on the RGB image and NIR image to obtain fused image data; Generating defect detection results and evaluating their confidence levels involves inputting fused image data into a trained defect detection model for feature extraction and target recognition, obtaining structured defect detection results that include defect location, type, and classification confidence, calculating the confidence level of the defect detection results, and comparing it with the confidence threshold to determine the reliability of the defect detection results and the appropriate processing path. Real-time inference of fused image data involves inputting the fused image data into a trained defect detection model for frame-by-frame analysis, combining it with the multi-scale feature extraction structure in the defect detection model to generate and analyze defect detection results. The defect detection model is expressed as: Among them, Y represents the defect detection result set, s represents the three output scales under multi-scale feature extraction, that is, the output resolution is 1 / 8, 1 / 16, and 1 / 32 of the input, ∪ represents the union of the output results under the three output scales, Neck s (.) indicates the fusion of features at the s-th scale, Backbone(.) indicates the lightweight detection backbone network, represents the set of network parameters trained at the sth scale, and F(x,y) represents the final grayscale value of the fused image at the pixel position (x,y); Evaluating the confidence of defect detection results includes constructing a confidence evaluation function based on the sample distribution characteristics, classification difficulty, and boundary features during the defect detection model training process, and comprehensively evaluating the defect location and defect type in combination with the spatial positioning matching degree and feature response strength of the potential defects. The confidence of the defect detection result is calculated and compared with the confidence threshold. If the confidence of the defect detection result is higher than the confidence threshold, the defect detection result is determined to be valid defect information, and the single board is marked as an abnormal single board. If the confidence level of the defect detection result is lower than the confidence threshold, the defect detection result is judged as invalid defect information and a re-inspection instruction is triggered; The confidence evaluation function is constructed as follows: Q(b (i) ,c (i) )Nα·D dist (c (i) )+β·E box (b (i) )+γ·S sample (c (i) ) Among them, Q(b (i) ,c (i) ) represents the confidence of the i-th defect detection result, α represents the classification difficulty item weight, D dist (.) represents the classification difficulty metric function, β represents the bounding box quality term weight, Ebox(.) represents the bounding box confidence function, γ represents the sample distribution sparsity term weight, S sample (.) represents the sample distribution scoring function; Comparing the confidence of the defect detection result with the confidence threshold is expressed as: Among them, Label (i) represents the judgment result of the i-th defect information, and δ represents the confidence threshold.

2. The method for detecting defects in an electric energy meter sheet based on multimodal data fusion according to claim 1, characterized in that: The synchronous acquisition of the electric energy meter board image and the fusion preprocessing include: capturing the visible light image of the electric energy meter board by an RGB camera to obtain an RGB image; capturing hidden defects by an NIR camera through the ink on the electric energy meter circuit board to obtain an NIR image; and preprocessing the RGB image and the NIR image.

3. The method for detecting defects in an electric energy meter sheet based on multimodal data fusion according to claim 2, wherein: Obtaining the fused image data includes calculating a fusion weight coefficient according to local contrast of the image, performing weighted fusion processing on the pre-processed RGB image and the NIR image, and obtaining the fused image data.

4. The method for detecting defects in an electric energy meter sheet based on multimodal data fusion according to claim 3, wherein: The training of the defect detection model includes constructing an image training set containing defect samples and normal samples, constructing a lightweight detection backbone network, training the defect detection model through the image training set, configuring a preset input size and optimization strategy, and optimizing the defect detection model parameters based on a preset round of training iterations.

5. The method for detecting defects in an electric energy meter sheet based on multimodal data fusion according to claim 4, characterized in that: The parsing of the defect detection results includes parsing the defect detection results output by the real-time reasoning of the defect detection model to obtain the defect location and defect type.

6. An electric energy meter panel defect detection system based on multimodal data fusion, which adopts the electric energy meter panel defect detection method based on multimodal data fusion according to any one of claims 1 to 5, characterized in that: Including fusion image processing module, neural network training module, defect detection and confidence assessment module; The fusion image processing module includes an image acquisition unit and a weighted fusion unit. The image acquisition unit is used to synchronously acquire multimodal image data of the electric energy meter panel through an RGB camera and an NIR camera. The weighted fusion unit is used to perform fusion preprocessing on the acquired image data. The neural network training module is used to construct a fusion image training set containing multiple types of defect samples and normal samples, perform multiple rounds of training and parameter updates based on the target detection loss function, and dynamically adjust the model structure and training strategy based on sample characteristics and training feedback results; The defect detection and confidence assessment module includes a defect reasoning unit and a confidence assessment unit. The defect reasoning unit is used to input the fused image data into the trained defect detection model and output the defect detection results. The confidence assessment unit is used to calculate the confidence index of each defect target in the model reasoning result, compare it with the preset threshold, output a high-confidence defect, or trigger a re-inspection mechanism.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the electric energy meter panel defect detection method based on multimodal data fusion according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the electric energy meter panel defect detection method based on multimodal data fusion according to any one of claims 1 to 5 are implemented.

Images