Refrigerant leakage detection method and device for recycling link of refrigeration equipment

Abstract

The invention discloses a refrigerant leakage detection method and device for a recovery link of refrigeration equipment. The method comprises the following steps: 1, acquiring a bimodal image; 2, identifying the foggy plume by using the visible light image; and 3, identifying a low-temperature abnormal region by using an infrared thermal image. And 4, mapping the bounding box on the visible light image to the infrared thermal imaging image, and calculating the area intersection-to-union ratio between the mapped bounding box and the low-temperature abnormal region. And 5, carrying out cross validation, and judging whether a refrigerant leakage event occurs or not. Visible light artifacts caused by dust and water vapor and infrared artifacts caused by environmental thermal noise in an industrial recovery environment are effectively inhibited by utilizing complementarity and non-correlation interference characteristics of visible light and infrared thermal imaging bimodes and through a combined strategy based on deep learning target detection, multi-mode cross validation and time sequence event aggregation; compared with a single-mode detection method and a conventional detection method, the method is higher in environmental adaptability and lower in false alarm rate.

Description

Technical Field

[0001] This invention belongs to the field of industrial safety and resource recycling technology, specifically relating to a method and device for detecting refrigerant leakage in the recycling process of refrigeration equipment. Background Technology

[0002] The standardized recycling and dismantling of used refrigeration equipment is a crucial link in environmental protection and resource recycling. Used electrical appliances containing refrigerant need to be properly disposed of, and the industrial environment in which they are located is subject to relevant laws and regulations. The scale and growth rate of this industry highlight the necessity of automated supervision. According to industry reports, in 2024, the number of recycled used air conditioners reached 15.85 million, a year-on-year increase of 72%, and the number of recycled used refrigerators reached 21.41 million, a year-on-year increase of 36.3%.

[0003] In industrial dismantling production lines, refrigerant must be recovered first during the dismantling and processing of refrigerators and air conditioners. However, due to improper drilling, violent operation, equipment aging, and other reasons during the refrigerant recovery process, refrigerant leakage is very likely to occur. Refrigerant leakage not only damages the environment but also poses risks such as frostbite and suffocation to operators. Therefore, supervising the recycling and dismantling workbench is an effective means to ensure industrial safety and efficient recycling.

[0004] Existing methods for refrigerant leak detection can be broadly categorized into external and internal detection. External detection: These methods determine refrigerant leaks by converting the leak into other characteristic signals. These include utilizing refrigerant flowability (soap solution detection), flame tests (halogen lamp detection), and changes in conductivity caused by the reaction of refrigerant with metal oxides (electronic sensors). Internal detection: These methods do not directly detect the refrigerant itself. This includes monitoring abnormal changes in refrigeration system operating parameters to determine the presence of leaks (based on real-time data), and establishing a normal operating model of the system and comparing actual and operating data to determine if a leak has occurred (based on actual physical modeling).

[0005] While the aforementioned methods are effective in routine maintenance, they are not ideal for monitoring open, dusty, foggy, and complex environments like "recycling and dismantling workstations," which require "full-scene, long-term, non-contact, and continuous coverage." However, the inventors discovered that refrigerant leaks at recycling workstations, due to the combined effects of the refrigerant's inherent properties and the Joule-Thomson effect, cause the surrounding air to cool rapidly during a leak, leading to condensation of ambient water vapor and the formation of a visible white, misty plume. This "visible plume" and the "low-temperature anomaly" constitute a natural dual-modal physical characterization, enabling dual-channel cross-validation and achieving highly robust, real-time, and comprehensive leak detection. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for detecting refrigerant leakage in the recycling process of refrigeration equipment.

[0007] In a first aspect, the present invention provides a method for detecting refrigerant leakage in the recycling process of refrigeration equipment, which includes continuously acquiring visible light images and infrared thermal images of the refrigeration equipment during the recycling operation.

[0008] The visible light image is input into the target detection model to identify the fog plume and generate the bounding box and confidence score of the fog plume.

[0009] Based on the temperature changes over time at different locations in the infrared thermal image and / or the temperature difference relative to the ambient temperature, low-temperature anomaly areas are extracted from the infrared thermal image.

[0010] Map the bounding box on the visible light image to the corresponding spatial location on the infrared thermal image, calculate the area crossover ratio (OCR) between the mapped bounding box and each low-temperature anomaly region in the infrared thermal image, and select the region with the largest OCR.

[0011] Cross-validation is performed based on the confidence level of the bounding box on the visible light image and the maximum area crossover ratio to determine whether a refrigerant leak event has occurred.

[0012] Preferably, the target detection model includes a multi-scale feature extraction module, a feature fusion and enhancement module, and a detection head. The multi-scale features output by the multi-scale feature extraction module are input into a high-frequency enhancement module within the feature fusion and enhancement module, where channel recalibration and local context modeling are performed on each scale feature. Subsequently, through a dynamic upsampling and spatial dependency awareness module, multi-path fusion is performed on the features at different scales. The resulting features are then smoothed by multi-scale convolution and output to the detection head.

[0013] Preferably, the highest-level features in the multi-scale features are processed by a deformable attention encoding module and then input into the high-frequency enhancement module. When performing self-attention calculation, the deformable attention encoding module introduces learnable offsets and sampling weights, adaptively selecting irregular sampling points related to the target region on the feature map. The dynamic upsampling includes a sampling point generator and a network sampling operator executed sequentially, adaptively generating position-related interpolation kernel weights through a lightweight sub-network. A skip connection is established between the input of the dynamic upsampling and the network sampling operator. The spatial dependency perception module includes a shallow feature extraction branch, a deep feature extraction branch, block-based attention feature extraction, and residual connections. The outputs of the shallow and deep feature extraction branches sequentially pass through the block-based attention feature extraction and residual connection modules. The output of the shallow feature extraction branch is skip-connected to the input of the residual connection module.

[0014] As a preferred method, the process of extracting low-temperature anomaly regions from infrared thermal images is as follows: A predetermined number of infrared thermal images acquired before the current moment are used as a sample set. The Euclidean distance between corresponding pixels in each sample set in the current infrared thermal image is calculated. Based on the obtained Euclidean distance, low-temperature anomaly pixels are selected from the current infrared thermal image; isolated noise points are eliminated, small holes are filled, and region boundaries are smoothed through morphological filtering to form the low-temperature anomaly region.

[0015] Preferably, the process of mapping the bounding box on the visible light image to its corresponding spatial position on the infrared thermal image is as follows: After the industrial camera and infrared thermal imager are installed and fixed, their intrinsic and extrinsic parameters are collected, and a spatial mapping relationship is established between the camera coordinate system and the infrared thermal imager coordinate system. The four corner points of the bounding box in the visible light image are then mapped onto the infrared thermal image using this spatial mapping relationship. The four corner points mapped onto the infrared thermal image are then connected sequentially to obtain the mapped bounding box.

[0016] As a preferred method, the process of cross-validation to determine a refrigerant leak event is as follows: when the confidence level of the identified mist plume bounding box is greater than the confidence threshold and the maximum area cross-union ratio is greater than the validation cross-union ratio threshold, a refrigerant leak event is determined to have occurred.

[0017] Preferably, the confidence threshold is 0.6 to 0.9; the crossover ratio threshold is in the range of 0.3 to 0.5.

[0018] Preferably, low-temperature anomaly regions are extracted from infrared thermal images acquired at the same time only when the confidence level of the bounding box of the fog-like plume is greater than the confidence level threshold.

[0019] Preferably, the image acquisition time corresponding to the refrigerant leak event is taken as the leak time. Visible light images acquired within a preset time range after the leak time are extracted, and each image is used for mist plume identification. The bounding box obtained from each visible light image is compared with the bounding box obtained from the visible light image of the previous frame using area intersection-union (IUU) calculation. If the number of IUU values ​​greater than the consecutive event IUU threshold exceeds a preset value, the leak event within the preset time range of the leak time is considered as a single continuous leak event, and the alarm is not triggered repeatedly.

[0020] Preferably, five visible light images are extracted after the leakage time. If at least three of the five visible light images have an inter-frame cross-union ratio (IU) greater than the continuous event IU threshold, the system determines that these continuous signals belong to the same continuous leakage event. The continuous event IU threshold is set to 0.6.

[0021] As a preferred approach, once a leak is identified as a post-aggregation leak, the system will automatically trigger an alarm and cache the continuous events and the visible light and infrared thermal images of the events before and after them for a preset duration, for subsequent accident analysis and tracing.

[0022] Preferably, the visible light image undergoes contrast adaptive histogram equalization processing before being input into the target detection model.

[0023] Secondly, the present invention provides a refrigerant leakage detection device for the recycling process of refrigeration equipment, which is used to perform the aforementioned refrigerant leakage detection method;

[0024] The refrigerant leak detection device includes an industrial camera, an infrared thermal imager, a data processing and recognition module, and a recycling and dismantling workbench. Both the industrial camera and the infrared thermal imager face the operating area of ​​the recycling and dismantling workbench. Visible light images and infrared thermal images acquired by the industrial camera and the infrared thermal imager are transmitted to the data processing and recognition module. The data processing and recognition module preprocesses the visible light images, identifies haze plumes, extracts low-temperature anomaly regions from the infrared thermal images, cross-validates the bounding boxes extracted from the visible light images with the low-temperature anomaly regions extracted from the infrared thermal images, and aggregates refrigerant leak detection results from different times.

[0025] Preferably, the industrial camera and the infrared thermal imager are mounted on the same bracket and are shockproofed so that their fields of view jointly cover the target workstation area.

[0026] Preferably, both the industrial camera and the infrared thermal imager have a frame rate of no less than 30fps to ensure sufficient capture of the transient leakage process.

[0027] The beneficial effects of this invention are:

[0028] 1. This invention establishes a dual-channel detection framework for visible light and infrared thermal imaging, and introduces a multimodal cross-validation decision mechanism with cross-union ratio (CUI) as its core. By complementaryly fusing the color and texture information of the visible light channel with the temperature difference information of the infrared channel, it effectively overcomes the problem of high false alarm rate of a single visible light channel under complex interference conditions such as dust and water vapor, while also compensating for the visual detection deficiencies of a single infrared thermal imaging channel under thermal noise interference and weak color and texture representation.

[0029] 2. This invention utilizes industrial cameras and infrared thermal imagers to acquire dual-modal image signals and, based on a target detection model for mist-like plumes, achieves "full-scene, long-term, non-contact, and continuous coverage" monitoring of refrigerant leaks at recycling stations. This effectively overcomes the limitations of traditional refrigerant detection methods (such as handheld detectors and point sensors) in terms of coverage and continuity, thereby significantly improving the safety supervision capabilities of the refrigeration equipment recycling process and ensuring the safety of industrial recycling operations.

[0030] 3. The target detection model in this invention introduces the extracted scale features into the high-frequency enhancement module to perform channel recalibration and local context modeling on each scale feature to enhance edge and detail response. Then, through the spatial dependency perception module, multi-path fusion of different scale features is performed from top to bottom and from bottom to top. While maintaining high-level semantic information, low-level spatial details are compensated, thereby improving the sensitivity to thin feather contours and weak texture regions.

[0031] 4. This invention utilizes a deformable attention mechanism to introduce learnable offsets and sampling weights when performing self-attention calculations. It can adaptively select irregular sampling points on the feature map that are highly correlated with the target region, thereby focusing more on key areas such as the boundary between the mist plume and the surrounding environment while performing global modeling, effectively suppressing the interference of large-area background and irrelevant textures on the detection results.

[0032] 5. In the top-down feature propagation path, this invention uses a dynamic upsampling module to replace the traditional nearest neighbor or bilinear interpolation upsampling. Through a lightweight sub-network, it adaptively generates position-related interpolation kernel weights based on the input features, enabling the upsampling process to perform more refined reconstruction of target boundaries and small structural regions, and reducing the problems of blurred small target features and loss of edge information caused by fixed interpolation kernels. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a refrigerant leak detection device provided in an embodiment of the present invention.

[0034] Figure 2 A flowchart of a refrigerant leak detection method provided in an embodiment of the present invention.

[0035] Figure 3 This is a network structure diagram of the target detection model in an embodiment of the present invention.

[0036] Figure 4 This is a network structure diagram of the deformable attention coding structure in an embodiment of the present invention.

[0037] Figure 5 This is a network structure diagram of the dynamic upsampling structure in an embodiment of the present invention.

[0038] Figure 6 This is a network structure diagram of the high-frequency enhancement module in an embodiment of the present invention.

[0039] Figure 7 This is a network structure diagram of the spatial dependency perception module in an embodiment of the present invention.

[0040] Figure 8 This is a diagram of a white, misty plume generated during a refrigerant leak.

[0041] Figure 9 This is a schematic diagram of the refrigerant leakage mist plume detection results according to an embodiment of the present invention.

[0042] Reference numerals: 1. Industrial camera; 2. Infrared thermal imager; 3. Bracket; 4. Data processing and identification module; 5. Refrigeration equipment to be recycled; 6. Recycling and dismantling workbench. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings.

[0044] like Figure 1 As shown, a refrigerant leak detection method for the recycling process of refrigeration equipment utilizes a refrigerant leak detection device comprising a multimodal sensor, a support 3, a data processing and identification module 4, and a recycling and dismantling workbench 6. Both the recycling and dismantling workbench 6 and the support 3 are fixed to the frame of the detection device. The multimodal sensor is fixed to the support 3 and positioned directly above the recycling and dismantling workbench 6. The image acquisition lens of the multimodal sensor faces the recycling and dismantling workbench 6, and the field of view of the multimodal sensor completely covers the operating area of ​​the recycling and dismantling workbench 6. During the detection process, the refrigeration equipment 5 to be recycled is placed on the operating area of ​​the recycling and dismantling workbench 6.

[0045] The multimodal sensor is used to acquire multimodal images and transmit them to the data processing and recognition module 4. In this embodiment, the multimodal sensor includes an industrial camera 1 and an infrared thermal imager 2 arranged side by side at the same height; both the industrial camera 1 and the infrared thermal imager 2 are downward-facing, directly facing the operating area of ​​the recycling and dismantling workbench 6. The industrial camera 1 is used to acquire visible light images of the refrigeration equipment 5 to be recycled; the infrared thermal imager 2 is used to acquire infrared thermal images of the refrigeration equipment 5 to be recycled. The data processing and recognition module 4 is used to preprocess the multimodal images, perform target detection, multimodal cross-validation, and event aggregation to achieve real-time monitoring and alarm of refrigerant leakage.

[0046] In this embodiment, the frame rate of both the industrial camera 1 and the infrared thermal imager 2 is no less than 30fps, thereby enabling sensitive capture of low-temperature anomalies caused by refrigerant leakage and ensuring that the dynamic process of the leakage moment can be captured.

[0047] like Figure 2 As shown, the refrigerant leak detection method includes the following steps:

[0048] Step 1: Calibration and Registration of Multimodal Sensors

[0049] 1-1. Due to the difference in the field of view between industrial camera 1 and infrared thermal imager 2, the visible light image and infrared thermal image generated by them do not naturally overlap in space, thus requiring geometric calibration and registration. During the system installation and debugging phase, a two-dimensional calibration board is placed in the operating area of ​​the recycling and dismantling workbench 6. The two-dimensional calibration board uses a checkerboard or dot array target. Multiple sets of calibration images are simultaneously acquired by industrial camera 1 and infrared thermal imager 2, and the intrinsic parameter matrices of industrial camera 1 and infrared thermal imager 2, as well as the extrinsic parameters relative to the world coordinate system, are obtained using Zhang's calibration method. The intrinsic parameter matrix includes focal length, principal point coordinates, and distortion coefficients.

[0050] 1-2. Using the coordinate system of industrial camera 1 as a reference, the relative rotation matrix and translation vector of infrared thermal imager 2 relative to industrial camera 1 are obtained from the extrinsic parameters of industrial camera 1 and infrared thermal imager 2, thus establishing a spatial mapping relationship from the visible light camera coordinate system to the infrared thermal imager coordinate system. Through this spatial mapping relationship, the pixel coordinates in the visible light image can be uniformly transformed to the pixel coordinate system of the infrared image, achieving spatial alignment of the two channels. Simultaneously, by setting a time synchronization mechanism or timestamp matching, the images acquired by industrial camera 1 and infrared thermal imager 2 correspond to the same moment in the leakage process in time, thereby completing the spatiotemporal registration of multimodal data.

[0051] This step provides the parameter basis for subsequent multimodal cross-validation and spatial overlap calculation.

[0052] Step 2: Image Acquisition and Preprocessing

[0053] 2-1. Multimodal Image Acquisition

[0054] The refrigeration equipment 5 to be recycled is placed on the operating area of ​​the recycling and dismantling workbench 6, and visible light images and infrared thermal images of the refrigeration equipment 5 to be recycled are collected by the industrial camera 1 and the infrared thermal imager 2, respectively.

[0055] 2-2. Visible Light Image Preprocessing

[0056] The visible light image is processed using the Contrast Adaptive Histogram Equalization (CLAHE) method, which effectively enhances the visual features corresponding to water vapor condensation, making them stand out more against complex industrial backgrounds. The visual features corresponding to water vapor condensation include the edge and texture features of the mist-like plumes formed by water vapor condensation.

[0057] Step 3: Target Detection

[0058] 3-1. Visible Light Target Detection

[0059] The visible light image obtained through step 2-2 is input into the pre-trained target detection model to obtain the visible light target detection results. The visible light target detection results include candidate bounding boxes representing the hazy plume and their corresponding confidence scores. In this embodiment, the target to be detected, the "white hazy plume," has characteristics of blurred edges, irregular shape, semi-transparency, and non-rigidity, and there is also a large amount of noise interference in the detection environment.

[0060] like Figure 3 As shown, the target detection model includes a multi-scale feature extraction module, a feature fusion enhancement module, and a detection head, all connected in sequence. The multi-scale feature extraction module comprises three convolutional normalization layers (ConvNormLAyer), one max-pooling layer, and four residual blocks (BasicBlock), all connected in sequence. The last three residual blocks output multi-scale features P5, P4, and P3, respectively, and are introduced into the feature fusion enhancement module. The residual blocks (BasicBlock) employ a stacked structure, extracting multi-scale feature maps with strides of 4, 8, 16, and 32 from bottom to top.

[0061] Given that the target "white misty plume" in this embodiment has characteristics such as blurred edges, irregular shape, semi-transparency, and non-rigidity, and that the detection environment is subject to a large amount of noise interference, this invention introduces the following improved modules into the benchmark RT-DETR model to enhance the detection capability for small-scale, low-contrast misty targets:

[0062] The feature fusion enhancement module introduces the multi-scale features P5, P4, and P3 output from the multi-scale feature extraction module into the high-frequency enhancement module HFP. Channel recalibration and local context modeling are performed on each scale feature to enhance edge and detail responses. Subsequently, the spatial dependency awareness module SDP performs multi-path fusion of different scale features from top to bottom and / or bottom to top, compensating for low-level spatial details while preserving high-level semantic information, thereby improving sensitivity to thin feather contours and weakly textured regions. Finally, after multi-scale convolutional smoothing, the resulting fused and enhanced three features are output to the detection head.

[0063] Specifically, the feature fusion enhancement module has three feature processing paths. In the first feature processing path, feature P5 passes through the deformable attention coding structure AIFI_DAttention and the high-frequency enhancement module HFP in sequence to obtain feature P5H; feature P5H is then convolved and output to the detection head.

[0064] In the second feature processing path, features P5 and P4 are fused layer by layer after passing through the high-frequency enhancement module HFP. Specifically, feature P4 passes through the high-frequency enhancement module HFP to obtain feature P4H; the dynamic upsampled features (DySample) of features P4H and P5H are fused through the spatial dependency perception module SDP, and the resulting feature is superimposed with the dynamic upsampled features of feature P5H to obtain feature P45; feature P45 is output to the detection head after convolution.

[0065] In the third feature processing path, features P5, P4, and P3 are fused layer by layer after passing through the high-frequency enhancement module HFP. Specifically, feature P3 passes through the high-frequency enhancement module HFP to obtain feature P3H; feature P5H is dynamically upsampled layer by layer, and after each layer of dynamic upsampling, features P4H and feature P3H are fused through the spatial dependency perception module SDP. The resulting feature is superimposed with the dynamically upsampled feature of feature P45 and then output to the detection head after convolution.

[0066] like Figure 4 As shown, the deformable attention encoding structure (AIFI_DAttention) includes the following steps executed sequentially: deformable attention mechanism feature extraction, layer normalization, 1×1 convolution, GELU activation function, dropout layer, 1×1 convolution, and deformable attention mechanism feature extraction. Skip connections are established before and after the first deformable attention mechanism feature extraction; a skip connection is also established between the output of layer normalization and the input of the second deformable attention mechanism feature extraction. When the deformable attention encoding structure performs self-attention computation, the deformable attention mechanism layer introduces learnable offsets and sampling weights, adaptively selecting irregular sampling points highly correlated with the target region on the feature map. This allows for greater focus on key regions such as the boundary between the mist plume and the surrounding environment, based on the global modeling of the standard attention encoder (AIFI), effectively suppressing the interference of large-area background and irrelevant textures on the detection results.

[0067] like Figure 5 As shown, the dynamic upsampling module in the feature fusion enhancement module includes a sampling point generator, a sampling point set, and a network sampling operator connected in sequence. A skip connection is established between the input and the network sampling operator. In the top-down feature propagation path, dynamic upsampling (DySample) replaces traditional nearest-neighbor or bilinear interpolation upsampling. Dynamic upsampling uses a lightweight sub-network to adaptively generate position-dependent interpolation kernel weights based on the input features, enabling the upsampling process to perform more refined reconstruction of target boundaries and small structural regions, mitigating the problems of blurred small target features and loss of edge information caused by fixed interpolation kernels.

[0068] like Figure 6As shown, the High Frequency Perception (HFP) module includes parallel spatial attention gating and channel attention gating, as well as a serial fusion module (Add), 3×3 convolution, and normalization (GroupNorm). The outputs of the spatial attention gating and channel attention gating are fused by the fusion module and then input into the 3×3 convolution. Both spatial and channel attention gating transform the feature maps to the frequency domain, thereby suppressing the background and enhancing the target edges. Specifically, the channel attention gating uses the high-frequency response map to determine which channels contain more target details, thereby calculating the corresponding weights to enhance the feature channels containing the target. The spatial attention gating uses the high-frequency response map to generate a spatial mask, allowing the model to focus spatially on edges and texture-rich regions.

[0069] like Figure 7 As shown, the Spatial Dependency Awareness Module (SDP) includes a shallow feature extraction branch, a deep feature extraction branch, a block-based attention feature extraction module, and a residual connection module. The outputs of the shallow and deep feature extraction branches are sequentially passed through the block-based attention feature extraction and residual connection modules. The output of the shallow feature extraction branch is skipped to the input of the residual connection module. The shallow feature extraction branch includes a shallow feature input layer, a 3×3 convolution, and a normalization (GroupNorm) connected in sequence; the deep feature extraction branch includes a deep feature input layer, a 3×3 convolution, and a normalization (GroupNorm) connected in sequence. The SDP corrects spatial misalignment issues during cross-scale fusion, and the corrected deep features can be accurately fused into the shallow features, greatly enriching the semantic information of small targets. Channel attention gating is also implemented.

[0070] In the detection head, features of different scales are unified to a fixed number of channels through convolutional projection, and global modeling is performed using a Transformer-based encoder-decoder structure to directly output the target category and bounding box regression results.

[0071] In some embodiments, the target detection model is built on the RT-DETR model, which uses a Transformer architecture and leverages its core self-attention mechanism to effectively capture global contextual information and long- and short-distance dependencies in the image. Since the target to be detected in this embodiment, the "white misty plume," has characteristics of blurred edges, irregular shape, semi-transparency, and non-rigidity, and the detection environment contains a large amount of noise interference, its core self-attention mechanism effectively captures global contextual information and long- and short-distance dependencies in the image. This is more conducive to understanding the correlation between the refrigerant leak phenomenon and surrounding refrigeration equipment and personnel, thus exhibiting a more significant detection advantage in this scenario compared to traditional CNN-based detectors (such as YOLO).

[0072] 3-2. Infrared Target Detection

[0073] Because the temperature at the refrigerant leak point is significantly lower than the ambient background due to the Joule-Thomson effect, a threshold segmentation and contour extraction algorithm is used. The process of the threshold segmentation and contour extraction algorithm is as follows: In an infrared thermal image, a decrease in temperature in a certain area will be represented by a decrease in the pixel value of that area; in this embodiment, the Vibe algorithm (Visual Background Extractor) is used to calculate the Euclidean distance between each pixel in the current infrared thermal image and the corresponding pixel in each sample in the sample set, thereby obtaining the low-temperature abnormal pixels in the current infrared thermal image.

[0074] The sample set includes infrared thermal images from a preset number of frames prior to the current time (the first 20 infrared thermal images in this embodiment). The current infrared thermal image is compared with each sample. For pixels at the same location in both the current infrared thermal image and each sample, if more than a preset number of sample pixels have a value greater than the pixel value in the current infrared thermal image, then that pixel in the current infrared thermal image is considered a low-temperature anomaly pixel. Low-temperature anomaly pixels are marked as foreground and assigned a pixel value of 255 (white), while the remaining pixels are marked as background and assigned a pixel value of 0 (black), thereby achieving dynamic background subtraction, effectively suppressing thermal noise interference in the environment, and greatly highlighting the dynamic low-temperature anomaly area caused by refrigerant leakage.

[0075] Morphological opening and closing operations are performed on the binarized segmentation mask obtained by the Vibe algorithm (in this embodiment, erosion-dilation and dilation-erosion operations are performed on 3×3 or 5×5 structuring elements) to remove isolated noise points, fill small holes and smooth region boundaries, thereby obtaining coherent low-temperature candidate regions.

[0076] Step 4: Multimodal Cross-validation

[0077] A real refrigerant leak will simultaneously generate a "visible plume" (visible light target detection result) and a "low temperature anomaly" (infrared target detection result), and the two overlap spatially to some extent. However, common interferences at the workstation (such as water vapor, smoke, and other heat sources) can only trigger a single-channel response. The specific process for calculating the spatial overlap between the visible light target detection results and the infrared target detection results is as follows:

[0078] 4-1. Using the intrinsic parameters, extrinsic parameters, and mapping matrix obtained in step one, map the points in the visible light image coordinate system to the infrared image coordinate system. For each candidate bounding box output by the visible light channel, take the pixel coordinates of its four corner points in the visible light image, map them to the infrared image coordinate system through the above mapping matrix, obtain the coordinates of the four vertices after mapping, and use them to construct the mapped visible light bounding box on the infrared image coordinate system.

[0079] 4-2. For the polygonal outline of each low-temperature anomaly region in the infrared channel, calculate the area intersection-union ratio (IoU) with the mapped visible light bounding box. The IoU can be expressed as the ratio of the area of ​​the intersection region of the visible light mapped bounding box region and the area of ​​the union region of the low-temperature anomaly region, as shown in the following expression:

[0080]

[0081] Among them, IoU i A represents the area intersection-union ratio corresponding to the i-th low-temperature anomaly region; B represents the visible light mapped bounding box region; i Let be the i-th independent low-temperature anomaly region. i = 1, 2, ..., m; m is the number of low-temperature anomaly regions.

[0082] 4-3. Take the maximum value among the area cross-union ratios corresponding to each low-temperature anomaly region as the maximum cross-union ratio. If the maximum cross-union ratio is greater than the preset verification cross-union ratio threshold, it is considered that the mist plume and the low-temperature anomaly region are highly overlapping in space, thereby realizing multimodal cross-verification and effectively distinguishing between real leakage events and single-channel interference.

[0083] Step 5: Decision Making and Alerts

[0084] A dual-judgment threshold is set for the final decision, effectively filtering false alarms. When the confidence level of the visible light target detection result is greater than the confidence threshold, and the maximum cross-union ratio of multimodal cross-validation is greater than the validation cross-union ratio threshold, a real leakage event is determined to have occurred, and an alarm is triggered.

[0085] In this embodiment, the confidence threshold is selected within the range of 0.6-0.9. When the confidence of the visible light target detection result is greater than the confidence threshold, it is considered a valid plume determination and enters the cross-validation process. The cross-union ratio (CUNR) threshold is selected within the range of 0.3-0.5. When the maximum CUNR is greater than the CUNR threshold, it indicates that the visible plume and the low-temperature anomaly area highly overlap in space, and only then is it considered a true, cross-validated refrigerant leak determination. After a leak is determined, the system will automatically cache multimodal images and detection data for a preset duration (e.g., 10 seconds before and after) before and after the event for subsequent accident analysis and tracing.

[0086] Step Six: Event Aggregation

[0087] The single-frame leakage event signals triggered in step five are aggregated along the time dimension to avoid misreporting a single continuous leakage event as multiple independent alarms. The system caches leakage event signals within a time window (e.g., 5 consecutive frames). When an event signal from step five is triggered, the system calculates the area intersection-union ratio (IU) between the current frame's bounding box and the previous frame's bounding box to determine the spatial continuity of the leakage source. Within the 5-frame time window, if at least 3 consecutive frames have an IU greater than 0.6, the system determines that these consecutive signals belong to a single continuous leakage event, triggering and maintaining only one alarm. After being identified as an aggregated leakage event, the system automatically triggers one alarm and caches the multimodal image and detection data of the continuous event for subsequent incident analysis and tracing.

[0088] The feasibility and engineering feasibility analysis of the refrigerant leakage detection method provided in this embodiment are as follows: (Considering the physical mechanism, sensor hardware requirements, algorithm implementation, and engineering deployment)

[0089] I. Feasibility at the level of physical mechanisms

[0090] Stable observability of bimodal characterization: on the one hand, such as Figure 8 As shown, the condensed white mist-like plume exhibits significant differences in brightness and texture in the visible light band, making it easily captured by industrial cameras. On the other hand, due to the inherent properties of the refrigerant, the ambient temperature around the leak area will drop dramatically, potentially causing frostbite to operators. A distinct temperature anomaly will form in the infrared band, easily distinguishable by infrared thermal imagers. Therefore, the "visible light + infrared" dual-channel detection framework constructed in this embodiment is highly compatible with the objective physical phenomenon of refrigerant leakage, ensuring the observability and distinguishability of the monitored object.

[0091] II. Feasibility of Sensors and Hardware Requirements

[0092] 1. Industrial cameras and infrared thermal imagers are mature industrial devices: Industrial cameras and infrared thermal imagers are widely used in industrial visual inspection, production line quality inspection, security monitoring, and equipment condition monitoring. They have mature product forms and complete industrial interfaces, and frame rates of 30fps and above are already standard specifications in existing industrial products. The resolution and frame rate parameters required in this embodiment are within the performance range of existing commercial equipment, without relying on new sensors that have not yet been industrialized.

[0093] 2. Installation method and site environment compatibility: This embodiment uses a shared bracket to place the industrial camera and infrared thermal imager above the recycling and dismantling workbench, providing overhead coverage of the work area. This installation method is basically consistent with the existing industrial vision system installation method on the production line. The bracket, protective cover, and shockproof structure can all be implemented using mature standard parts or simple customized components, without fundamentally changing the existing recycling workstation layout, and has good engineering feasibility.

[0094] 3. Computing and storage resources are readily available: The image preprocessing, target detection, and simple geometric transformation operations required in this embodiment can be completed within milliseconds using current mainstream industrial computing units (such as industrial PCs, embedded GPU modules, edge computing boxes, etc.). Combined with an acquisition frame rate of approximately 30fps, it can meet the requirements for real-time monitoring at the recycling station; the image and detection data caching for several seconds before and after the event also means that the storage bandwidth and capacity requirements are within the tolerance range of existing industrial storage devices.

[0095] III. Feasibility of Algorithm and Software Implementation

[0096] 1. Mature and usable preprocessing algorithms: The CLAHE contrast adaptive histogram equalization algorithm and ViBe dynamic background modeling algorithm used in this invention are both mature and publicly available methods in the field of computer vision. There are abundant engineering implementations and open source code resources available. They can be directly called or easily reproduced in commonly used vision libraries such as OpenCV. The algorithm complexity is moderate and suitable for real-time operation in industrial settings.

[0097] 2. Effectiveness of the Base Target Detection Model: This invention improves upon the RT-DETR-r18 benchmark model with HS-FPN feature enhancement, AIFI-DA attention deformable attention structure, and DySample dynamic upsampling. The structure is clear, the parameter scale is controllable, and it can be trained and deployed using existing deep learning frameworks (such as PyTorch and TensorRT). It does not introduce complex structures that are difficult to implement in this field or rely on specialized hardware. Existing detection results, such as… Figure 9 As shown.

[0098] 3. Model training data can be constructed: The data of this invention comes from partner companies, and first-hand real monitoring videos can be obtained. It is possible to construct a dataset of refrigerant leakage in the dismantling process of waste refrigeration equipment. Those skilled in the art can complete the data collection and annotation work based on conventional experimental conditions, and there is no infeasibility in data acquisition.

[0099] 4. The multimodal registration and IoU calculation process is simple and reliable: The Zhang calibration method, homography matrix solution, and bounding box-polygon area intersection-union ratio calculation adopted in this invention are all conventional techniques in the fields of computer vision and geometric computing, with extensive engineering practice and mature library functions (such as the camera calibration and geometric transformation module in OpenCV, and polygon intersection-union area calculation in common geometric libraries). Therefore, there are no technical obstacles in the software implementation of the spatial mapping and intersection-union ratio calculation process for multimodal images.

[0100] IV. Feasibility of Project Deployment and Operation / Maintenance

[0101] 1. Good compatibility with existing recycling processes: The detection device of this invention monitors the recycling station in a "non-contact, top-down" manner, without requiring substantial modifications to existing refrigerant recovery equipment, tooling fixtures, or dismantling processes; it only requires the installation of a multimodal sensor bracket above the station and the connection of the data processing and identification module to the existing power supply and network system, without significantly affecting the work cycle.

[0102] 2. System parameters can be optimized on-site: Key parameters such as confidence threshold, cross-union ratio (CURRR) threshold, and consecutive frame count threshold can all be debugged and optimized based on on-site trial operation data. The parameter value ranges provided by this invention (such as confidence 0.6-0.9, CURRR 0.3-0.5, and consecutive frame window of 5 frames) are easy to implement and adjust in engineering, which is beneficial for customized configuration according to the noise levels and working habits of different enterprises and workstations.

[0103] 3. Low maintenance cost and strong scalability: On the one hand, the maintenance of industrial cameras and infrared thermal imagers mainly includes routine work such as lens cleaning, protective cover inspection, and fastening of fixed structures. The maintenance cycle is long and the cost is low. On the other hand, the detection algorithm is deployed in the computing unit and can be iteratively optimized through software upgrades. This makes it easy to add new scenarios and categories or connect to the enterprise's upper-level system to realize functions such as alarm linkage, data cloud uploading, and remote monitoring.

[0104] The above description, in conjunction with specific examples, provides further explanation of the invention and should not be construed as limiting the specific implementation of the invention to these descriptions. For those skilled in the art, simple deductions or substitutions can be made without departing from the inventive concept, and all such modifications and substitutions should be considered within the scope of protection of this invention.

Claims

1. A method for detecting refrigerant leakage in the refrigerant recovery process of refrigeration equipment, characterized in that: Visible light images and infrared thermal images of the refrigeration equipment are continuously collected during the refrigeration equipment recovery operation; The visible light image is input into the target detection model to identify the fog plume and generate the bounding box and confidence score of the fog plume. Based on the temperature changes over time at different locations in the infrared thermal image and / or the temperature difference relative to the ambient temperature, low-temperature anomaly areas are extracted from the infrared thermal image. Map the bounding box on the visible light image to the corresponding spatial position on the infrared thermal image, calculate the area intersection-union ratio between the mapped bounding box and each low temperature anomaly region in the infrared thermal image, and select the region with the largest area intersection-union ratio. Cross-validation is performed based on the confidence level of the bounding box on the visible light image and the maximum area crossover ratio to determine whether a refrigerant leak event has occurred.

2. The refrigerant leakage detection method according to claim 1, characterized in that: The target detection model includes a multi-scale feature extraction module, a feature fusion enhancement module, and a detection head. The multi-scale features output by the multi-scale feature extraction module are input into the high-frequency enhancement module in the feature fusion enhancement module to perform channel recalibration and local context modeling on each scale feature. Then, through the dynamic upsampling and spatial dependency perception module, the features of different scales are fused in multiple paths. The resulting features are output to the detection head after being smoothed by multi-scale convolution.

3. The refrigerant leakage detection method according to claim 2, characterized in that: The highest-level features in the multi-scale features are processed by the deformable attention encoding module and then input into the high-frequency enhancement module. When the deformable attention encoding module performs self-attention calculation, it introduces learnable offsets and sampling weights to adaptively select irregular sampling points related to the target region on the feature map. The dynamic upsampling includes a sampling point generator and a network sampling operator executed sequentially, and a position-related interpolation kernel weight is adaptively generated through a lightweight sub-network. There is a skip connection between the input of dynamic upsampling and the network sampling operator. The spatial dependency perception module includes a shallow feature extraction branch, a deep feature extraction branch, a block attention feature extraction, and a residual connection. The outputs of the shallow feature extraction branch and the deep feature extraction branch are sequentially processed by the block attention feature extraction and residual connection modules. The output of the shallow feature extraction branch is skipped to the input of the residual connection module.

4. The refrigerant leakage detection method according to claim 1, characterized in that: The process of extracting low-temperature anomaly regions from infrared thermal images is as follows: a preset number of infrared thermal images collected before the current moment are used as a sample set. Calculate the Euclidean distance between corresponding pixels in the sample set of the current infrared thermal image; based on the obtained Euclidean distance, filter out low-temperature abnormal pixels in the current infrared thermal image; Low-temperature anomaly regions are formed by morphological filtering.

5. The refrigerant leakage detection method according to claim 1, characterized in that: The process of mapping the bounding box on the visible light image to its corresponding spatial position on the infrared thermal image is as follows: collect the intrinsic and extrinsic parameters of the industrial camera and infrared thermal imager after installation and fixation, and establish the spatial mapping relationship from the optical camera coordinate system to the infrared thermal imager coordinate system; map the four corner points of the bounding box in the visible light image to the infrared thermal image through the spatial mapping relationship; and connect the four corner points obtained by mapping on the infrared thermal image in sequence to obtain the mapped bounding box.

6. The refrigerant leakage detection method according to claim 1, characterized in that: The process of cross-validation to determine a refrigerant leak event is as follows: when the confidence level of the identified mist plume bounding box is greater than the confidence threshold, and the maximum area cross-union ratio is greater than the validation cross-union ratio threshold, a refrigerant leak event is determined to have occurred.

7. The refrigerant leakage detection method according to claim 6, characterized in that: Low-temperature anomaly regions are extracted from infrared thermal images acquired at the same time only when the confidence level of the bounding box of the fog-like plume is greater than the confidence level threshold.

8. The refrigerant leakage detection method according to claim 1, characterized in that: The image acquisition time corresponding to the refrigerant leak event is taken as the leak time; visible light images acquired within a preset time range after the leak time are extracted and fog-like plume identification is performed respectively; the bounding box obtained from each frame of visible light image is compared with the bounding box obtained from the visible light image of the previous frame to calculate the area intersection-union ratio; if the number of area intersection-union ratios greater than the consecutive event intersection-union ratio threshold is greater than a preset value, the leak event within the preset time range of the leak time is regarded as the same continuous leak event, and the alarm is not triggered repeatedly.

9. The refrigerant leakage detection method according to claim 1, characterized in that: The visible light image undergoes contrast adaptive histogram equalization before being input into the target detection model.

10. A refrigerant leakage detection device for the refrigerant recovery process in refrigeration equipment, characterized in that: Used to perform the refrigerant leak detection method as described in claim 9; The refrigerant leak detection device includes an industrial camera (1), an infrared thermal imager (2), a data processing and identification module (4), and a recycling and dismantling workbench (6). The industrial camera (1) and the infrared thermal imager (2) are both oriented towards the operating area of ​​the recycling and dismantling workbench (6). The visible light images and infrared thermal images collected by the industrial camera (1) and the infrared thermal imager (2) are transmitted to the data processing and identification module (4). The data processing and identification module (4) preprocesses the visible light images, identifies the fog plume, extracts the low-temperature abnormal area from the infrared thermal image, cross-validates the bounding box extracted from the visible light images with the low-temperature abnormal area extracted from the infrared thermal image, and aggregates the refrigerant leak detection results at different times.

Images