Method and device for calibrating bounding box based on visible light and infrared image decision-level fusion detection

Abstract

The application discloses a visible light and infrared image decision-level fusion detection frame calibration method and device, and relates to the technical field of sensor calibration. The method comprises the following steps: performing distortion removal processing on the obtained visible light image and infrared image; obtaining the distortion-removed visible light image and infrared image and performing manual labeling to obtain the parameters of the visible light image labeled frame and the parameters of the infrared image labeled frame; respectively processing the distortion-removed visible light image and infrared image based on the parameters of the labeled frame to obtain a visible light-infrared registration image, and inputting the registration image into a YOLOv5 algorithm to obtain detection frame parameters; according to the detection frame parameters, respectively calculating the detection frame parameters of the visible light detection frame parameters converted to the infrared coordinate system and the detection frame parameters of the infrared detection frame parameters converted to the visible light coordinate system; and performing decision-level fusion on the converted visible light detection frame parameters and infrared detection frame parameters to obtain a fusion result. The application can reduce calibration errors and increase calibration speed.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, and in particular to a method and apparatus for calibrating a detection frame based on decision-level fusion of visible light and infrared images. Background Technology

[0002] Target detection technology plays a crucial role in security monitoring, intelligent transportation, autonomous driving, drones, and the military. These applications rely on target detection systems to accurately identify and locate target objects in a scene in real time, providing decision support and ensuring safety. With the rapid development of artificial intelligence, computer vision, and sensor technology, target detection technology is gradually shifting from single-modality to multi-modal fusion, especially the fusion detection of visible light and infrared sensors. Traditional target detection often relies on a single type of sensor, such as a visible light camera; however, a single sensor is limited by lighting, weather, and occlusion factors, and cannot provide sufficient reliability in all environments. The fusion of visible light and infrared sensors can effectively overcome the limitations of a single sensor. Visible light sensors can capture rich color information and details, making them suitable for operation under normal lighting conditions, while infrared sensors acquire information based on thermal radiation, exhibiting good adaptability in low light, nighttime, or adverse weather conditions such as fog, haze, rain, and snow. By fusing images from two different modalities, target detection systems can obtain more comprehensive and robust target detection capabilities, especially in complex environments and extreme conditions, improving target detectability and recognition accuracy.

[0003] However, images from different modalities, such as visible light images and infrared images, often suffer from spatial or geometric misalignment due to differences in acquisition conditions, sensor characteristics, and shooting angles. This means that even if two images depict the same scene, the information they contain does not directly correspond spatially. Therefore, a simple fusion of two images is not possible, presenting a challenge in image fusion: how to effectively register images from different sensors to ensure they are fused in the same coordinate system. The core issue of this registration lies in the fact that different external parameters of the sensors (such as position, viewing angle, and rotation) lead to geometric errors in the two images. To correctly fuse visible light and infrared images, precise calibration and alignment are essential. This process typically involves solving for the spatial relationship between the two sensors, thereby transforming the coordinates of the two images so they can be processed and fused in a unified coordinate system. Therefore, the key to the fusion problem is aligning image information from different sources into a common coordinate system. This alignment process requires comprehensive consideration of sensor geometric differences, optical characteristics, and shooting angle factors. By employing high-precision calibration methods and image registration techniques, the spatial inconsistency problem of images with different modalities can be solved, ensuring that they can be accurately fused, thereby providing reliable data support for subsequent target detection, tracking, and recognition tasks.

[0004] Currently, the calibration methods for visible light and infrared images include: (1) Manual calibration based on calibration plates: using a specific calibration plate (such as a checkerboard) to collect visible light and infrared images from different perspectives, and calculating the geometric transformation relationship between sensors by extracting corner points or feature points in the images. This method is relatively accurate, but requires tedious manual operation and calibration plates. For infrared sensors, calibration plates made of special materials are usually required; (2) Automatic calibration based on image matching: using feature point matching (such as SIFT and SURF algorithms) or epipolar constraints, etc., to automatically find corresponding points between visible light and infrared images, and calculate the extrinsic parameters (rotation matrix and displacement vector) of the sensor through the relationship between these points. This method does not require a calibration plate, but has high requirements for image quality and the number of matching points, especially since infrared images usually have low resolution and the automatic calibration error is large; (3) Deep learning: without obtaining the sensor extrinsic parameters, data fusion between sensors is directly achieved through extensive empirical learning. By training a deep neural network model, the spatial transformation relationship between visible light and infrared images is directly learned from the image pairs. Deep learning models automatically learn the correspondence between images by using a large amount of labeled data (image pairs), avoiding the process of manually extracting feature points in traditional labeling methods. However, the disadvantages are that they require a large amount of data and model training, have high requirements for data volume and computer hardware, and the algorithms are usually extremely complex and difficult to learn. Summary of the Invention

[0005] To address the technical problems of cumbersome manual operation, large calibration errors, and the need for large amounts of data for intensive training of deep learning algorithms in existing technologies, this invention provides a method and apparatus for calibration of detection boxes based on decision-level fusion of visible light and infrared images. The technical solution is as follows:

[0006] On the one hand, a method for calibration of detection boxes based on decision-level fusion of visible light and infrared images is provided. This method is implemented by a detection box calibration device based on decision-level fusion of visible light and infrared images, and includes:

[0007] S1. Obtain visible light images, infrared images, and intrinsic parameters and distortion coefficients of the visible light camera and infrared camera; based on the intrinsic parameters and distortion coefficients, perform distortion correction processing on the visible light image and infrared image respectively to obtain distorted visible light images and distorted infrared images.

[0008] S2. Based on the distorted visible light image and the distorted infrared image, obtain the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image; based on the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image, calculate the aspect ratio factor between the distorted visible light image and the distorted infrared image.

[0009] S3. Based on the aspect ratio factor, calculate the width and height of the distorted visible light image after scaling and scale the visible light image to obtain a scaled visible light image; crop the scaled visible light image to an image with the same resolution as the distorted infrared image to obtain a cropped visible light image; calculate the position of the cropped visible light image in the distorted visible light image to obtain two visible light-infrared registration images with the same resolution and aligned pixel positions.

[0010] S4. Input the visible light-infrared registered images into the YOLOv5 target detection algorithm to obtain the detection box parameters of the visible light image and the detection box parameters of the infrared image respectively.

[0011] S5. Based on the detection box parameters of the visible light image, calculate the coordinates and size information of the detection box in the infrared image coordinate system to obtain the detection box parameters of the transformed visible light image; perform decision-level fusion based on the detection box parameters of the transformed visible light image and the detection box parameters of the infrared image to obtain the final fusion result.

[0012] On the other hand, a detection box calibration device based on visible light and infrared image decision-level fusion is provided. This device is applied to the detection box calibration method based on visible light and infrared image decision-level fusion. The device includes:

[0013] The processing unit is used to acquire visible light images, infrared images, and intrinsic parameters and distortion coefficients of visible light cameras and infrared cameras; based on the intrinsic parameters and distortion coefficients, it performs distortion correction processing on the visible light images and infrared images respectively to obtain distorted visible light images and distorted infrared images.

[0014] The calculation unit is used to obtain parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image based on the distorted visible light image and the distorted infrared image; and to calculate the aspect ratio factor between the distorted visible light image and the distorted infrared image based on the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image.

[0015] The acquisition unit is configured to calculate the scaled width and height of the distorted visible light image based on the aspect ratio factor, and scale the visible light image to obtain a scaled visible light image; crop the scaled visible light image to an image with the same resolution as the distorted infrared image to obtain a cropped visible light image; calculate the position of the cropped visible light image in the distorted visible light image to obtain two visible light-infrared registration images with the same resolution and aligned pixel positions.

[0016] The detection unit is used to input the visible light-infrared registered images into the YOLOv5 target detection algorithm to obtain the detection box parameters of the visible light image and the detection box parameters of the infrared image, respectively.

[0017] The fusion unit is used to calculate the coordinates and size information of the detection box in the visible light image transformed into the infrared image coordinate system based on the detection box parameters of the visible light image, and obtain the detection box parameters of the transformed visible light image; and to perform decision-level fusion based on the detection box parameters of the transformed visible light image and the detection box parameters of the infrared image to obtain the final fusion result.

[0018] On the other hand, a detection box calibration device based on visible light and infrared image decision-level fusion is provided. The detection box calibration device based on visible light and infrared image decision-level fusion includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods described above for detection box calibration based on visible light and infrared image decision-level fusion is implemented.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for decision-level fusion detection box calibration based on visible light and infrared images.

[0020] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0021] This invention eliminates the need for customized infrared calibration boards, precision turntables, or additional synchronization triggers. Calibration can be completed using any on-site "person-object" target, resulting in near-zero deployment costs. Compared to traditional calibration methods, this invention's calibration method is shorter in duration and requires less data. It only needs one frame of visible light, one frame of infrared light, and two bounding boxes to calculate global mapping parameters. It is unaffected by resolution, texture, lighting, or weather conditions, and can still operate stably even with blurred infrared images or overexposed visible light.

[0022] The image registration error obtained using the calibration method of this invention is ≤1.5 pixels, meeting the requirements of engineering-level fusion detection. The same set of (K_W, K_H, ΔX, ΔY) quadruple parameters can be used for image-level pixel / feature fusion as well as for decision-level bounding box coordinate transformation, and can also be reverse-mapped, flexibly supporting bidirectional fusion. After camera disassembly, focal length changes, and baseline shifts, on-site re-image taking, annotation, and parameter recalculation are sufficient, eliminating the need for factory return or model retraining, resulting in a short maintenance cycle and low computational cost. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a detection box calibration method based on decision-level fusion of visible light and infrared images provided by an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of a detection frame calibration method for visible light-infrared registration provided by an embodiment of the present invention;

[0026] Figure 3 This is a flowchart of a detection box calibration method provided in an embodiment of the present invention for decision-level detection box fusion in visible light-infrared fusion detection;

[0027] Figure 4 This is a block diagram of a detection box calibration device based on decision-level fusion of visible light and infrared images provided in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of a detection frame calibration device based on decision-level fusion of visible light and infrared images provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0030] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0031] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0032] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0033] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0034] This invention provides a method for calibration of detection boxes based on decision-level fusion of visible light and infrared images. This method can be implemented by a calibration device based on decision-level fusion of visible light and infrared images, which can be a terminal or a server. Figure 1 The flowchart shown is for a detection box calibration method based on decision-level fusion of visible light and infrared images. The processing flow of this method may include the following steps:

[0035] S1. Obtain the visible light image, infrared image, and intrinsic parameters and distortion coefficients of the visible light camera and infrared camera; based on the intrinsic parameters and distortion coefficients, perform distortion correction processing on the visible light image and infrared image respectively to obtain the distortion-corrected visible light image and the distortion-corrected infrared image.

[0036] In one feasible implementation, a visible light camera and an infrared camera are mounted side by side, with their relative positions fixed. The visible light camera has a field of view of 64°×40°, and the infrared camera has a field of view of 48.1°×38.4°. During installation, it is ensured that the field of view of the two cameras have a large overlap so that the images output by the two sensors can see the common parts, thereby preparing for subsequent calibration and fusion.

[0037] The visible light camera and infrared camera are connected to the PC via Ethernet. To ensure that visible light and infrared images can be acquired simultaneously, NTP time synchronization is first performed on each device. The camera's ROS2 driver is then started, and the images of the two modalities with the closest timestamps are obtained through ROS2 topic parsing, thus achieving time alignment. Therefore, 1920 images can be acquired in real time. 1080 visible light image and 640 Infrared image with a resolution of 512.

[0038] Among them, a checkerboard calibration was used to obtain the intrinsic parameters and distortion coefficients of the visible light camera and the infrared camera.

[0039] in, Figure 2 This is a flowchart of a detection frame calibration method for visible light-infrared registration provided by an embodiment of the present invention.

[0040] S2. Based on the distorted visible light image and the distorted infrared image, obtain the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image; based on the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image, calculate the aspect ratio factor between the distorted visible light image and the distorted infrared image.

[0041] Optionally, the aspect ratio factor between the distorted visible light image and the distorted infrared image is expressed by the following formulas (1)-(2):

[0042] K_W=W ir / W vi (1)

[0043] K_H=H ir / W ir (2)

[0044] Where K_W represents the width scaling factor between the distorted visible light image and the distorted infrared image; K_H represents the height scaling factor between the distorted visible light image and the distorted infrared image; W ir Indicates the pixel width of the infrared bounding box; W vi H represents the pixel width of the visible light bounding box. ir This indicates the pixel height of the infrared bounding box.

[0045] In one feasible implementation, targets are manually labeled in both the distortion-corrected visible light and infrared images, and the parameter information of the respective bounding boxes is recorded simultaneously. The parameter information of the visible light image bounding box is represented as (X... vi ,Y viW vi H vi The parameter information of the infrared image bounding box is represented as (X). ir ,Y ir W ir H ir ); where X vi This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image; Y... vi This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image; W vi H represents the pixel width of the visible light bounding box. vi The visible light bounding box represents the pixel height; Xir represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image; Y... ir This indicates the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image; W ir H represents the pixel width of the infrared bounding box. ir This indicates the pixel height of the infrared bounding box.

[0046] S3. Based on the aspect ratio factor, calculate the width and height of the distorted visible light image after scaling and scale the visible light image to obtain the scaled visible light image; crop the scaled visible light image to an image with the same resolution as the distorted infrared image to obtain the cropped visible light image; calculate the position of the cropped visible light image in the distorted visible light image to obtain two visible light-infrared registration images with the same resolution and aligned pixel positions.

[0047] Through the scaling process described above, it is possible to obtain areas where the visible light image and the infrared image can directly overlap at the pixel level. That is, by further cropping, the two images can be approximately aligned.

[0048] In one feasible implementation, the scaled visible light image is further cropped to the same resolution as the infrared image, thereby achieving pixel-level alignment of coordinates. Specifically, the infrared image is used as a template to crop an image of the same resolution from the visible light image. Since pixel alignment is required, the cropping coordinates need to be calculated using calibration parameters as a reference. For a more intuitive explanation, the infrared image is superimposed on the visible light image. When the calibration boxes of the infrared and visible light images completely overlap, the size of the rectangle occupied by the infrared image is the size of the cropping box of the visible light image.

[0049] Specifically, based on the parameter information of the visible light image bounding box and the infrared image bounding box, the visible light and infrared images are scaled and transformed to ensure that the two modal images maintain the same pixel ratio, thus preparing for subsequent cropping operations.

[0050] Optionally, the process of S3 calculating the scaled width and height of the distorted visible light image based on the aspect ratio factor is expressed by the following formulas (3)-(4):

[0051] (3)

[0052] (4)

[0053] in, This represents the width of the distorted visible light image after scaling. Indicates the width of the distorted visible light image; The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This represents the height of the distorted visible light image after scaling. Indicates the height of the distorted visible light image; This represents the high scaling factor between the distorted visible light image and the distorted infrared image.

[0054] Optionally, the calculated position of the cropped visible light image in the distortion-free visible light image in S3 includes:

[0055] S31. Obtain the width and height of the cropped visible light image; wherein the width and height of the cropped visible light image are consistent with the height and width of the distortion-free infrared image;

[0056] S32. Based on the width and height scaling factors between the distortion-corrected visible light image and the distortion-corrected infrared image, calculate the parameter information of the scaled visible cursor bounding box.

[0057] Optionally, the parameter information of the scaled visible cursor bounding box is represented by the following formulas (5)-(6):

[0058] (5)

[0059] (6)

[0060] in, This indicates the x-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This indicates the y-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. This represents the high scaling factor between the distorted visible light image and the distorted infrared image.

[0061] S33. Based on the parameter information of the scaled visible cursor bounding box, calculate the x and y coordinates of the top left corner of the cropping box;

[0062] Alternatively, the x and y coordinates of the top left corner of the cropping frame are represented by the following formulas (7)-(8):

[0063] (7)

[0064] (8)

[0065] in, This represents the x-coordinate of the top-left corner of the cropping box; This indicates the pixel x-coordinate deviation between the new calibration box and the infrared image detection box; This indicates the x-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. This represents the y-coordinate of the top-left corner of the cropping box; This indicates the pixel y-coordinate deviation between the new calibration box and the infrared image detection box; This indicates the y-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image.

[0066] S34. Based on the width and height of the cropped visible light image and the x and y coordinates of the upper left corner of the cropping frame, obtain the position of the cropped visible light image in the distortion-free visible light image.

[0067] S4. Input the visible light-infrared registered images into the YOLOv5 target detection algorithm to obtain the detection box parameters of the visible light image and the infrared image, respectively.

[0068] S5. Based on the detection box parameters of the visible light image, calculate the coordinates and size information of the detection box in the infrared image coordinate system to obtain the detection box parameters of the transformed visible light image; perform decision-level fusion based on the detection box parameters of the transformed visible light image and the detection box parameters of the infrared image to obtain the final fusion result.

[0069] Among them, such as Figure 3 The diagram shows a flowchart of a detection box calibration method provided by an embodiment of the present invention for decision-level detection box fusion in visible light-infrared fusion detection.

[0070] Optionally, the specific implementation process of S5 includes: S51-S56:

[0071] S51. Based on the aspect ratio factor between the distorted visible light image and the distorted infrared image, calculate the x-coordinate deviation and y-coordinate deviation of the visible light image detection box transformed into the infrared coordinate system, expressed by the following formulas (9)-(10):

[0072] (9)

[0073] (10)

[0074] in, This indicates the x-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. This represents the width scaling factor between visible light and infrared images; This represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. This indicates the y-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. This indicates the high scaling factor between visible light and infrared images; This represents the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image.

[0075] S52. Based on the x-coordinate deviation and y-coordinate deviation of the visible light image detection box transformed into the infrared coordinate system, calculate the x-coordinate and y-coordinate of the detection box in the infrared image coordinate system, and express them by the following formulas (11)-(12):

[0076] (11)

[0077] (12)

[0078] in, This represents the x-coordinate of the detection box in the visible light image coordinate system relative to the infrared image coordinate system. This represents the x-coordinate of the detection box in the visible light image coordinate system. This indicates the x-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the y-coordinate of the detection box in the visible light image coordinate system relative to the infrared image coordinate system. This represents the y-coordinate of the detection box in the visible light image coordinate system. This indicates the y-coordinate deviation when the visible light image detection box is transformed into the infrared coordinate system.

[0079] S53. Based on the target detection box parameter information of the distorted visible light image and the width and height scaling factor between the distorted visible light image and the distorted infrared image, calculate the width and height of the visible light detection box transformed into the infrared image coordinate system, and express them by the following formulas (13)-(14):

[0080] (13)

[0081] (14)

[0082] in, This represents the width of the visible light detection frame when transformed to the infrared coordinate system; This represents the width of the visible light detection box obtained from the visible light image inferred by YOLOv5; The width scaling factor representing the difference between the distorted visible light image and the distorted infrared image; The height scaling factor between visible light and infrared images; This indicates the height of the visible light detection frame when converted to the infrared coordinate system; This represents the height of the visible light detection box obtained from the visible light image inferred by YOLOv5. This indicates the high scaling factor between the distorted visible light image and the distorted infrared image;

[0083] S54. Based on the x and y coordinates of the visible light image detection box in the infrared image coordinate system and the width and height of the visible light detection box after transformation to the infrared image coordinate system, obtain the parameters of the visible light detection box in the infrared image coordinate system.

[0084] S55. Based on the above steps S51-S54, calculate the coordinates and size information of the infrared detection box in the visible light image coordinate system to obtain the detection box parameters of the transformed infrared image.

[0085] S56. Decision-level fusion is performed based on the detection box parameters of the converted visible light image and the detection box parameters of the infrared image to obtain the final fusion result; or decision-level fusion is performed based on the detection box parameters of the converted infrared image and the detection box parameters of the visible light image to obtain the final fusion result.

[0086] In one feasible implementation, the calculation process for the infrared detection frame parameters is the same as that for the visible light detection frame parameters.

[0087] This invention addresses the spatial registration problem and is applicable to all three fusion methods for visible-infrared multimodal fusion detection. Specifically, it can be used for both image registration and bounding box fusion tasks. While image registration is a necessary step in visible-infrared image fusion, the calibration method proposed in this invention makes image registration optional in decision-level fusion, allowing bounding box calibration and fusion to proceed directly without image registration.

[0088] This invention eliminates the need for customized infrared calibration boards, precision turntables, or additional synchronization triggers. Calibration can be completed using any on-site "person-object" target, resulting in near-zero deployment costs. Compared to traditional calibration methods, this invention's calibration method is shorter in duration and requires less data. It only needs one frame of visible light, one frame of infrared light, and two bounding boxes to calculate global mapping parameters. It is unaffected by resolution, texture, lighting, or weather conditions, and can still operate stably even with blurred infrared images or overexposed visible light.

[0089] The image registration error obtained using the calibration method of this invention is ≤1.5 pixels, meeting the requirements of engineering-level fusion detection. The same set of (K_W, K_H, ΔX, ΔY) quadruple parameters can be used for image-level pixel / feature fusion as well as for decision-level bounding box coordinate transformation, and can also be reverse-mapped, flexibly supporting bidirectional fusion. After camera disassembly, focal length changes, and baseline shifts, on-site re-image taking, annotation, and parameter recalculation are sufficient, eliminating the need for factory return or model retraining, resulting in a short maintenance cycle and low computational cost.

[0090] Figure 4 This is a block diagram of a detection box calibration device based on decision-level fusion of visible light and infrared images, provided in an embodiment of the present invention. This device is used for a detection box calibration method based on decision-level fusion of visible light and infrared images. (Refer to...) Figure 4 The device includes a processing unit 410, a computing unit 420, an acquisition unit 430, a detection unit 440, and a fusion unit 450. Wherein:

[0091] The processing unit 410 is used to acquire visible light images, infrared images, and intrinsic parameters and distortion coefficients of visible light cameras and infrared cameras; based on the intrinsic parameters and distortion coefficients, it performs distortion correction processing on the visible light images and infrared images respectively to obtain distorted visible light images and distorted infrared images.

[0092] The calculation unit 420 is used to obtain parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image based on the distorted visible light image and the distorted infrared image; and to calculate the aspect ratio factor between the distorted visible light image and the distorted infrared image based on the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image.

[0093] The acquisition unit 430 is used to calculate the width and height of the distorted visible light image after scaling according to the aspect ratio factor, and to scale the visible light image to obtain a scaled visible light image; to crop the scaled visible light image into an image with the same resolution as the distorted infrared image to obtain a cropped visible light image; and to calculate the position of the cropped visible light image in the distorted visible light image to obtain two visible light-infrared registration images with the same resolution and aligned pixel positions.

[0094] The detection unit 440 is used to input the visible light-infrared registered images into the YOLOv5 target detection algorithm to obtain the detection box parameters of the visible light image and the detection box parameters of the infrared image, respectively.

[0095] The fusion unit 450 is used to calculate the coordinates and size information of the detection box in the visible light image transformed into the infrared image coordinate system based on the detection box parameters of the visible light image, and obtain the detection box parameters of the transformed visible light image; and to perform decision-level fusion based on the detection box parameters of the transformed visible light image and the detection box parameters of the infrared image to obtain the final fusion result.

[0096] Optionally, the aspect ratio factor between the distorted visible light image and the distorted infrared image is expressed by the following formulas (1)-(2):

[0097] K_W=W ir / W vi (1)

[0098] K_H=H ir / W ir (2)

[0099] Where K_W represents the width scaling factor between the distorted visible light image and the distorted infrared image; K_H represents the height scaling factor between the distorted visible light image and the distorted infrared image; W ir Indicates the pixel width of the infrared bounding box; W vi H represents the pixel width of the visible light bounding box. ir This indicates the pixel height of the infrared bounding box.

[0100] Optionally, the process of calculating the scaled width and height of the distortion-free visible light image based on the aspect ratio factor is represented by the following formulas (3)-(4):

[0101] (3)

[0102] (4)

[0103] in, This represents the width of the distorted visible light image after scaling. Indicates the width of the distorted visible light image; The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This represents the height of the distorted visible light image after scaling. Indicates the height of the distorted visible light image; This represents the high scaling factor between the distorted visible light image and the distorted infrared image.

[0104] Optionally, calculating the position of the cropped visible light image within the distortion-free visible light image includes:

[0105] Obtain the width and height of the cropped visible light image; wherein the width and height of the cropped visible light image are consistent with the height and width of the infrared image;

[0106] Based on the width and height scaling factors between the distortion-corrected visible light image and the distortion-corrected infrared image, calculate the parameter information of the scaled visible cursor bounding box.

[0107] Based on the parameter information of the scaled visible cursor bounding box, calculate the x and y coordinates of the top left corner of the cropping box;

[0108] Based on the width and height of the cropped visible light image and the x and y coordinates of the top left corner of the cropping frame, the position of the cropped visible light image within the distorted visible light image is obtained.

[0109] Optionally, the parameter information of the scaled visible cursor bounding box is represented by the following formulas (5)-(6):

[0110] (5)

[0111] (6)

[0112] in, This indicates the x-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This indicates the y-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. This represents the high scaling factor between the distorted visible light image and the distorted infrared image.

[0113] Optionally, the x and y coordinates of the upper left corner of the cropping frame are represented by the following formulas (7)-(8):

[0114] (7)

[0115] (8)

[0116] in, This represents the x-coordinate of the top-left corner of the cropping box; This indicates the pixel x-coordinate deviation between the new calibration box and the infrared image detection box; This indicates the x-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. This represents the y-coordinate of the top-left corner of the cropping box; This indicates the pixel y-coordinate deviation between the new calibration box and the infrared image detection box; This indicates the y-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image.

[0117] Optionally, the fusion unit 450 is used for:

[0118] (1) Based on the aspect ratio factor between the distortion-free visible light image and the distortion-free infrared image, calculate the x-coordinate deviation and y-coordinate deviation of the visible light image detection box transformed into the infrared coordinate system, which are expressed by the following formulas (9)-(10):

[0119] (9)

[0120] (10)

[0121] in, This indicates the x-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The width scaling factor between the distorted visible light image and the infrared image; This represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. This indicates the y-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The height scaling factor between the distorted visible light image and the distorted infrared image; This represents the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image.

[0122] (2) Based on the x-coordinate deviation and y-coordinate deviation of the visible light image detection box transformed into the infrared coordinate system, calculate the x-coordinate and y-coordinate of the detection box in the infrared image coordinate system, and express them by the following formulas (11)-(12):

[0123] (11)

[0124] (12)

[0125] in, This represents the x-coordinate of the detection box in the visible light image coordinate system relative to the infrared image coordinate system. This represents the x-coordinate of the detection box in the visible light image coordinate system. This indicates the x-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the y-coordinate of the detection box in the visible light image coordinate system relative to the infrared image coordinate system. This represents the y-coordinate of the detection box in the visible light image coordinate system. This indicates the y-coordinate deviation when the visible light image detection box is transformed into the infrared coordinate system.

[0126] (3) Based on the target detection box parameter information of the distorted visible light image and the width and height scaling factors between the distorted visible light image and the distorted infrared image, calculate the width and height of the visible light detection box transformed into the infrared image coordinate system, and express them by the following formulas (13)-(14):

[0127] (13)

[0128] (14)

[0129] in, This represents the width of the visible light detection frame when transformed to the infrared coordinate system; This represents the width of the visible light detection box obtained from the visible light image inferred by YOLOv5; The width scaling factor representing the difference between the distorted visible light image and the distorted infrared image; The height scaling factor between visible light and infrared images; This indicates the height of the visible light detection frame when converted to the infrared coordinate system; This represents the height of the visible light detection box obtained from the visible light image inferred by YOLOv5. The height scaling factor between the distorted visible light image and the distorted infrared image;

[0130] (4) Based on the x and y coordinates of the visible light image detection box in the infrared image coordinate system and the width and height of the visible light detection box after transformation to the infrared image coordinate system, obtain the parameters of the visible light detection box in the infrared image coordinate system after transformation.

[0131] (5) Based on the above steps (1)-(4), calculate the coordinates and size information of the infrared detection box in the visible light image coordinate system to obtain the detection box parameters of the transformed infrared image;

[0132] (6) Decision-level fusion is performed based on the detection box parameters of the converted visible light image and the detection box parameters of the infrared image to obtain the final fusion result; or decision-level fusion is performed based on the detection box parameters of the converted infrared image and the detection box parameters of the visible light image to obtain the final fusion result.

[0133] This invention eliminates the need for customized infrared calibration boards, precision turntables, or additional synchronization triggers. Calibration can be completed using any on-site "person-object" target, resulting in near-zero deployment costs. Compared to traditional calibration methods, this invention's calibration method is shorter in duration and requires less data. It only needs one frame of visible light, one frame of infrared light, and two bounding boxes to calculate global mapping parameters. It is unaffected by resolution, texture, lighting, or weather conditions, and can still operate stably even with blurred infrared images or overexposed visible light.

[0134] The image registration error obtained using the calibration method of this invention is ≤1.5 pixels, meeting the requirements of engineering-level fusion detection. The same set of (K_W, K_H, ΔX, ΔY) quadruple parameters can be used for image-level pixel / feature fusion as well as for decision-level bounding box coordinate transformation, and can also be reverse-mapped, flexibly supporting bidirectional fusion. After camera disassembly, focal length changes, and baseline shifts, on-site re-image taking, annotation, and parameter recalculation are sufficient, eliminating the need for factory return or model retraining, resulting in a short maintenance cycle and low computational cost.

[0135] Figure 5 This is a schematic diagram of a detection box calibration device based on decision-level fusion of visible light and infrared images provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the detection box calibration device based on visible light and infrared image decision-level fusion can include the above-mentioned... Figure 4 The illustrated detection box calibration device is based on visible light and infrared image decision-level fusion. Optionally, the visible light and infrared image decision-level fusion detection box calibration device 510 may include a first processor 2001.

[0136] Optionally, the visible light and infrared image-based decision-level fusion detection frame calibration device 510 may also include a memory 2002 and a transceiver 2003.

[0137] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0138] The following is combined with Figure 5 A detailed description of each component of the visible light and infrared image-based decision-level fusion detection frame calibration device 510 is provided below:

[0139] The first processor 2001 is the control center of the visible light and infrared image decision-level fusion detection frame calibration device 510. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0140] Optionally, the first processor 2001 can perform various functions of the visible light and infrared image decision-level fusion detection frame calibration device 510 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0141] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0142] In a specific implementation, as one example, the visible light and infrared image-based decision-level fusion detection box calibration device 510 may also include multiple processors, for example... Figure 5 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0143] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0144] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the visible light and infrared image-based decision-level fusion detection frame calibration device 510. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0145] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0146] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0147] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the visible light and infrared image decision-level fusion detection frame calibration device 510. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0148] It should be noted that, Figure 5 The structure of the visible light and infrared image decision-level fusion detection frame calibration device 510 shown in the figure does not constitute a limitation on the router. The actual visible light and infrared image decision-level fusion detection frame calibration device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0149] Furthermore, the technical effect of the visible light and infrared image decision-level fusion detection box calibration device 510 can be referred to the technical effect of the visible light and infrared image decision-level fusion detection box calibration method described in the above method embodiments, and will not be repeated here.

[0150] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0151] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0152] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0153] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0154] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0155] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0156] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0158] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0161] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for calibration of detection boxes based on decision-level fusion of visible light and infrared images, characterized in that, The method includes: S1. Obtain visible light images, infrared images, and intrinsic parameters and distortion coefficients of the visible light camera and infrared camera; based on the intrinsic parameters and distortion coefficients, perform distortion correction processing on the visible light image and infrared image respectively to obtain distorted visible light images and distorted infrared images. S2. Based on the distorted visible light image and the distorted infrared image, obtain the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image; based on the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image, calculate the aspect ratio factor between the distorted visible light image and the distorted infrared image. S3. Based on the aspect ratio factor, calculate the width and height of the distorted visible light image after scaling and scale the visible light image to obtain a scaled visible light image; crop the scaled visible light image to an image with the same resolution as the distorted infrared image to obtain a cropped visible light image; calculate the position of the cropped visible light image in the distorted visible light image to obtain two visible light-infrared registration images with the same resolution and aligned pixel positions. The process of S3 calculating the width and height of the distorted visible light image after scaling based on the aspect ratio factor is represented by the following formulas (1)-(2): （1） （2） in, This represents the width of the distorted visible light image after scaling. Indicates the width of the distorted visible light image; The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This represents the height of the distorted visible light image after scaling. Indicates the height of the distorted visible light image; This represents the high scaling factor between the distorted visible light image and the distorted infrared image. S4. Input the visible light-infrared registered images into the YOLOv5 target detection algorithm to obtain the detection box parameters of the visible light image and the detection box parameters of the infrared image respectively. S5. Based on the detection box parameters of the visible light image, calculate the coordinates and size information of the detection box in the infrared image coordinate system to obtain the detection box parameters of the transformed visible light image; perform decision-level fusion based on the detection box parameters of the transformed visible light image and the detection box parameters of the infrared image to obtain the final fusion result.

2. The detection box calibration method based on decision-level fusion of visible light and infrared images according to claim 1, characterized in that, The aspect ratio factor between the distorted visible light image and the distorted infrared image is expressed by the following formulas (3)-(4): K_W= W vi / W ir (3) K_H=H vi / H ir (4) Where K_W represents the width scaling factor between the distorted visible light image and the distorted infrared image; K_H represents the height scaling factor between the distorted visible light image and the distorted infrared image; W ir Indicates the pixel width of the infrared bounding box; W vi H represents the pixel width of the visible light bounding box. ir H represents the pixel height of the infrared bounding box. vi This represents the pixel height of the visible light bounding box.

3. The detection box calibration method based on decision-level fusion of visible light and infrared images according to claim 1, characterized in that, The calculation of the position of the cropped visible light image in the distortion-free visible light image in S3 includes: S31. Obtain the width and height of the cropped visible light image; wherein the width and height of the cropped visible light image are consistent with the height and width of the distortion-free infrared image; S32. Based on the width and height scaling factors between the distortion-corrected visible light image and the distortion-corrected infrared image, calculate the parameter information of the scaled visible cursor bounding box. S33. Based on the parameter information of the scaled visible cursor bounding box, calculate the x and y coordinates of the top left corner of the cropping box; S34. Based on the width and height of the cropped visible light image and the x and y coordinates of the upper left corner of the cropping frame, obtain the position of the cropped visible light image in the distortion-free visible light image.

4. The detection box calibration method based on decision-level fusion of visible light and infrared images according to claim 3, characterized in that, The parameter information of the scaled visible cursor bounding box is expressed by the following formulas (5)-(6): （5） （6） in, This indicates the x-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This indicates the y-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. This represents the high scaling factor between the distorted visible light image and the distorted infrared image.

5. The detection box calibration method based on decision-level fusion of visible light and infrared images according to claim 3, characterized in that, The x and y coordinates of the upper left corner of the cropping frame are represented by the following formulas (7)-(8): （7） （8） in, This represents the x-coordinate of the top-left corner of the cropping box; This indicates the pixel x-coordinate deviation between the new calibration box and the infrared image detection box; This indicates the x-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. This represents the y-coordinate of the top-left corner of the cropping box; This indicates the pixel y-coordinate deviation between the new calibration box and the infrared image detection box; This indicates the y-coordinate of the top-left corner of the new calibration box of the scaled visible light image; This represents the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image.

6. The detection box calibration method based on decision-level fusion of visible light and infrared images according to claim 1, characterized in that, The detection box parameters based on the visible light image in S5 are obtained by calculating the coordinates and size information of the detection box in the visible light image transformed into the infrared image coordinate system. Decision-level fusion is performed based on the bounding box parameters of the converted visible light image and the bounding box parameters of the infrared image to obtain the final fusion result, including: S51. Based on the width and height scaling factors between the distorted visible light image and the distorted infrared image, calculate the x-coordinate deviation and y-coordinate deviation of the visible light image detection box transformed into the infrared coordinate system, expressed by the following formulas (9)-(10): （9） （10） in, This indicates the x-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the X-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The width scaling factor represents the difference between the distorted visible light image and the distorted infrared image. This represents the X-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. This indicates the y-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the Y-axis pixel coordinates of the top-left corner of the visible light bounding box in the image. The height scaling factor between the distorted visible light image and the distorted infrared image; This represents the Y-axis pixel coordinates of the top-left corner of the infrared bounding box in the image. S52. Based on the x-coordinate deviation and y-coordinate deviation of the visible light image detection box transformed into the infrared coordinate system, calculate the x-coordinate and y-coordinate of the detection box in the infrared image coordinate system, and express them by the following formulas (11)-(12): （11） （12） in, This represents the x-coordinate of the detection box in the visible light image coordinate system relative to the infrared image coordinate system. This represents the x-coordinate of the detection box in the visible light image coordinate system. This indicates the x-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; This represents the y-coordinate of the detection box in the visible light image coordinate system relative to the infrared image coordinate system. This represents the y-coordinate of the detection box in the visible light image coordinate system. This indicates the y-coordinate deviation when the visible light image detection box is transformed to the infrared coordinate system; S53. Based on the target detection box parameter information of the distorted visible light image and the width and height scaling factor between the distorted visible light image and the distorted infrared image, calculate the width and height of the visible light detection box transformed into the infrared image coordinate system, and express them by the following formulas (13)-(14): （13） （14） in, This represents the width of the visible light detection frame when transformed to the infrared coordinate system; This represents the width of the visible light detection box obtained from the visible light image inferred by YOLOv5; This represents the width scaling factor between visible light and infrared images; The height scaling factor between the distorted visible light image and the distorted infrared image; This indicates the height of the visible light detection frame when converted to the infrared coordinate system; This represents the height of the visible light detection box obtained from the visible light image inferred by YOLOv5. This indicates the high scaling factor between visible light and infrared images; S54. Based on the x and y coordinates of the visible light image detection box in the infrared image coordinate system and the width and height of the visible light detection box after transformation to the infrared image coordinate system, obtain the detection box parameters of the visible light image in the infrared image coordinate system. S55. Based on the above steps S51-S54, calculate the coordinates and size information of the infrared detection box in the visible light image coordinate system to obtain the detection box parameters of the transformed infrared image. S56. Decision-level fusion is performed based on the detection box parameters of the converted visible light image and the detection box parameters of the infrared image to obtain the final fusion result; or decision-level fusion is performed based on the detection box parameters of the converted infrared image and the detection box parameters of the visible light image to obtain the final fusion result.

7. A detection box calibration device based on visible light and infrared image decision-level fusion, wherein the detection box calibration device is used to implement the detection box calibration method based on visible light and infrared image decision-level fusion as described in any one of claims 1-6, characterized in that, The device includes: The processing unit is used to acquire visible light images, infrared images, and intrinsic parameters and distortion coefficients of visible light cameras and infrared cameras; based on the intrinsic parameters and distortion coefficients, it performs distortion correction processing on the visible light images and infrared images respectively to obtain distorted visible light images and distorted infrared images. The calculation unit is used to obtain parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image based on the distorted visible light image and the distorted infrared image; and to calculate the aspect ratio factor between the distorted visible light image and the distorted infrared image based on the parameter information of the bounding box of the distorted visible light image and the bounding box of the distorted infrared image. The acquisition unit is configured to calculate the scaled width and height of the distorted visible light image based on the aspect ratio factor, and scale the visible light image to obtain a scaled visible light image; crop the scaled visible light image to an image with the same resolution as the distorted infrared image to obtain a cropped visible light image; calculate the position of the cropped visible light image in the distorted visible light image to obtain two visible light-infrared registration images with the same resolution and aligned pixel positions. The detection unit is used to input the visible light-infrared registered images into the YOLOv5 target detection algorithm to obtain the detection box parameters of the visible light image and the detection box parameters of the infrared image, respectively. The fusion unit is used to calculate the coordinates and size information of the detection box in the visible light image transformed into the infrared image coordinate system based on the detection box parameters of the visible light image, and obtain the detection box parameters of the transformed visible light image; and to perform decision-level fusion based on the detection box parameters of the transformed visible light image and the detection box parameters of the infrared image to obtain the final fusion result.

8. A detection box calibration device based on decision-level fusion of visible light and infrared images, characterized in that, The detection box calibration device based on visible light and infrared image decision-level fusion includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images