An infrared channel label generation method suitable for a dual-channel camera

Abstract

The application discloses an infrared channel label generation method suitable for a dual-channel camera, and the method is characterized in that: a mapping relationship from a visible light target coordinate to an infrared target coordinate is constructed through internal parameters and distortion parameters of a visible light camera and an infrared camera, and in particular, through inverse transformation of the distortion parameters of the infrared camera, the corrected overlapping area coordinate is converted into a distortion coordinate of an original infrared image, and finally, a target detection label conforming to the characteristics of the infrared image is generated. The application can significantly reduce the labeling cost of the infrared image, improve the training efficiency and precision of an infrared target detection model, and is suitable for security, monitoring, automatic driving and other fields based on the dual-channel camera.

Description

Technical Field

[0001] This invention relates to a method for generating infrared channel tags applicable to dual-channel cameras, belonging to the field of computer vision and target detection technology. Background Technology

[0002] Dual-channel cameras (comprising both visible light and infrared lenses) are widely used in security monitoring, autonomous driving, and industrial inspection because they can detect targets under complex lighting conditions (such as at night or in fog). In target detection tasks, model training relies on a large amount of labeled data, but infrared images suffer from low resolution and blurred target features, leading to high costs, low efficiency, and difficulty in guaranteeing labeling accuracy through manual annotation.

[0003] In existing technologies, some methods attempt to directly map visible light tags onto infrared images by utilizing the overlapping field of view of visible light and infrared images from dual-channel cameras. However, this approach does not fully consider the differences in optical distortion between the two cameras. Due to differences in lens optical characteristics, visible light and infrared cameras exhibit their own radial and tangential distortions. Direct mapping can lead to significant deviations between the infrared tag and the actual target position. Furthermore, the overlapping field of view of dual-channel cameras is typically defined based on the corrected, distortion-free image, while the original data of the infrared image still retains distortion. Ignoring the inverse transformation from corrected coordinates to original distorted coordinates will further reduce tag accuracy.

[0004] In summary, existing methods for generating infrared channel labels for dual-channel cameras suffer from low label accuracy and high manual labeling costs due to the failure to consider the distortion characteristics of each dual-channel camera. There is an urgent need for a method that can accurately handle the differences in optical distortion between dual-channel cameras and achieve accurate conversion from visible light labels to infrared labels in order to solve the problem of infrared image labeling. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose an infrared channel tag generation method applicable to dual-channel cameras. By utilizing the existing target tags in the visible light channel of the dual-channel camera and through coordinate transformation and distortion correction, the target tags in the infrared channel are automatically generated, and the visible light tags are accurately mapped to the infrared channel.

[0006] The technical solution of this invention is: A method for generating infrared channel tags applicable to dual-channel cameras, comprising: Acquire the visible light target detection tags acquired by the dual-channel camera's visible light camera, the intrinsic parameters and distortion coefficients of the visible light camera and the infrared camera, the original image size, and the coordinate range of the overlapping area of ​​the visible light and infrared fields of view; The normalized coordinates of the visible light target detection label are converted into pixel coordinates of the visible light image to obtain the coordinates of the four corner points of the target bounding box; based on the intrinsic parameter matrix and distortion coefficients of the visible light camera, distortion correction is performed on the visible light image to obtain the corrected visible light image and the coordinates of the four corner points of the target bounding box in the image. Based on the intrinsic parameter matrix and distortion coefficients of the infrared camera, distortion correction is performed on the infrared image to obtain the corrected infrared image. Based on the coordinate range of the overlapping area between the visible light and infrared fields of view, the area overlapping with the infrared field of view is cropped from the corrected visible light image, and the coordinates of the four corner points of the target bounding box are restricted to the overlapping area; based on the size ratio of the overlapping area and the infrared corrected image, the target coordinates in the overlapping area are mapped to the infrared corrected coordinate system. Using the intrinsic parameter matrix and distortion coefficients of the infrared camera, inverse distortion calculation is performed on the infrared-corrected coordinates to obtain the coordinates of the four corner points of the target bounding box in the original infrared image, and then an infrared target detection label conforming to the target detection format is generated.

[0007] Furthermore, inverse distortion calculations are performed on the infrared-corrected coordinates: 1) Adjust the coordinates after infrared correction Convert to normalized coordinates : ,

[0008] In the formula, , , , These are the parameters of the infrared camera's intrinsic parameter matrix; 2) Calculate the radial distortion factor and tangential distortion compensation based on the distortion coefficient of the infrared camera;

[0009] In the formula, The radial distortion coefficient of the infrared camera. The sum of squares of the normalized coordinates. Radial distortion factor;

[0010] In the formula, This represents the tangential distortion coefficient of the infrared camera. , For the distortion coefficient Directional compensation amount, Directional compensation amount; 3) Based on the radial distortion factor and the tangential distortion compensation amount, the normalized distortion coordinates of the original infrared image are obtained, and then converted into the pixel coordinates of the original infrared image.

[0011] Furthermore, based on the radial distortion factor and the tangential distortion compensation amount, the normalized distortion coordinates of the original infrared image are obtained. :

[0012] Convert the normalized distortion coordinates to pixel coordinates of the original infrared image:

[0013] In the formula, These are the pixel coordinates of the original infrared image.

[0014] Furthermore, based on the coordinates of the four corner points of the target bounding box in the original infrared image, infrared target detection labels conforming to the target detection format are generated, wherein: Calculate the center coordinates:

[0015] Calculate the width:

[0016] Calculate height:

[0017] In the formula, the coordinates of the four corner points of the target bounding box in the original infrared image are respectively , , , ; Based on the center coordinates, width, and height, the data is normalized to the original infrared image size to generate infrared target detection tags in YOLO format.

[0018] Furthermore, if the visible light image acquired by the visible light camera is not a full-frame image, the visible light image is enlarged to the original image size, and the coordinates of the four corner points of the target bounding box are mapped into the enlarged image; then, based on the intrinsic parameter matrix and distortion coefficients of the visible light camera, distortion correction is performed on the enlarged image to obtain the corrected visible light image and the coordinates of the four corner points of the target bounding box in the image.

[0019] Furthermore, if the infrared image captured by the infrared camera is not a full-frame image, the infrared image is enlarged to the original image size, and then the distortion is corrected on the enlarged image based on the intrinsic parameter matrix and distortion coefficient of the infrared camera to obtain the corrected infrared image.

[0020] Furthermore, based on the size ratio of the overlapping region to the infrared-corrected image, the target coordinates within the overlapping region are mapped to the infrared-corrected coordinate system, specifically as follows: The coordinates of the visible light target confined within the overlapping region are mapped to the coordinate system of the infrared-corrected image. During mapping, the overlapping region of the visible light-corrected image and the infrared-corrected image correspond completely in space, differing only in size. Let the coordinates of the upper left corner of the overlapping region in the corrected visible light image be... The coordinates of the lower right corner are The width of the overlapping region is The height is ; For any point within the overlapping region of the visible light corrected image The corresponding coordinates in the distortion-free coordinate system of the infrared-corrected image for: ,

[0021] In the formula, the width of the image after infrared correction is Height is .

[0022] Furthermore, based on the intrinsic parameter matrix of the visible light camera and distortion coefficient The distortion correction function is called to correct the distortion of the four corner points of the target bounding box in the expanded image.

[0023] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0024] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0025] The advantages of this invention compared to the prior art are: (1) High precision: This invention processes the distortion characteristics of visible light and infrared cameras separately, especially performs inverse distortion transformation on the infrared-corrected coordinates, so that the generated infrared tag can accurately match the target position in the original infrared image, thus solving the tag offset problem caused by the optical difference of dual-channel cameras.

[0026] (2) Low cost: This invention does not require manual labeling of infrared images, but directly uses visible light tags to generate infrared tags, which significantly reduces the manpower and time costs of data labeling.

[0027] (3) Strong versatility: This invention is applicable to various dual-channel camera devices. Only the intrinsic parameters and distortion parameters of the corresponding camera need to be input to realize label conversion. It can be directly applied to the training of target detection models based on infrared images to improve model performance. Attached Figure Description

[0028] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating the method for generating infrared channel tags using a dual-channel camera, applicable to embodiments of the present invention. Figure 2 The images shown are the visible light images after being expanded to their original output size according to an embodiment of the present invention. (a) is the uncorrected image, and (b) is the corrected image. Figure 3 The images shown are infrared images from an embodiment of the present invention. (a) is an uncorrected image, and (b) is a corrected image. Figure 4 An overlapping region map is determined for embodiments of the present invention. Detailed Implementation

[0029] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0030] This invention proposes an infrared channel tag generation method applicable to dual-channel cameras, suitable for scenarios requiring target detection using both visible light and infrared images.

[0031] This method is as follows Figure 1 As shown, it includes: (1) Obtain basic data and parameters Obtain target detection labels from images captured by the visible light lens of a dual-channel camera. The label format is based on the visible light image (size: [size missing]). Target detection labels (e.g., in YOLO format, including target category, normalized center coordinates, width, and height); Obtaining visible light camera parameters: intrinsic parameter matrix Distortion coefficient (including radial distortion coefficient) , , and tangential distortion coefficient , ), visible light original image size; Obtaining Infrared Camera Parameters: Intrinsic Parameter Matrix Distortion coefficient (including radial distortion coefficient) , , and tangential distortion coefficient , ), and the original infrared image (size) ); Determine the coordinate range of the overlapping region of the dual-channel camera's field of view in the visible light-corrected image (coordinates of the top left corner of the overlapping region). lower right corner coordinates ).

[0032] (2) Visible light tag coordinate transformation and correction First, the normalized coordinates of the visible light label are converted into pixel coordinates of the visible light image to obtain the coordinates of the four corner points of the target bounding box; let the width of the visible light image be... Height is Normalized center coordinates and normalized width and height The corresponding pixel coordinates are: ,

[0033] The coordinates of the four corner points of the target bounding box (top left, top right, bottom left, bottom right) are: ,

[0034] ,

[0035] It is important to note that the size of the original visible light image output by the detector may differ from the size of the displayed image. In this method, the displayed or processed visible light image is the central region of the original detector output, with edge pixels truncated, while the infrared image is the full image of the original detector output. To correctly perform distortion correction, the truncated edge regions must be padded with black borders to restore the original detector output size. Therefore, the current visible light image is enlarged, with an enlarged width of [missing information]. The height is Let the expansion offset be... Then the original coordinates of the four corner points The coordinates in the augmented image are: ,

[0036] Subsequently, the intrinsic parameter matrix of the visible light camera was used. and distortion coefficient Distortion correction is performed on the four corner points of the target bounding box in the augmented image. The correction process uses a standard distortion model: first, the pixel coordinates are converted to normalized coordinates; then, the distortion-free normalized coordinates are iteratively solved using radial and tangential distortion models; finally, the pixel coordinates are converted back to the corrected values. This method directly calls a general distortion correction function (such as cv2.undistortPoints) to perform the above calculations, obtaining the coordinates of the four corner points of the target bounding box in the corrected visible light image. The same distortion correction method is used to correct the infrared image as well, and the size of the corrected infrared image remains unchanged, with a width of [missing value]. Height is .

[0037] (3) Coordinate mapping of overlapping regions Based on the calibration results of the overlapping region of the dual-camera fields of view, it is known that the coordinates of the upper left corner of the overlapping region in the corrected visible light image are as follows: The coordinates of the lower right corner are The coordinates of the four corner points of the corrected visible light target bounding box obtained in the previous step are restricted to this overlapping area: if a coordinate exceeds the range of the overlapping area, the boundary value is taken.

[0038] The coordinates of the visible light target, confined within the overlapping region, are mapped to the coordinate system of the infrared-corrected image. During mapping, the overlapping region of the visible light-corrected image and the infrared-corrected image correspond completely in space, differing only in size. Let the width of the overlapping region be... The height is Then for any point within the overlapping region... Its corresponding coordinates in the infrared-corrected image for: ,

[0039] This gives us the position of the target in the undistorted coordinate system after infrared correction.

[0040] (4) Inverse distortion transformation of infrared coordinates Using the intrinsic parameter matrix of an infrared camera and distortion coefficient The inverse distortion calculation is performed on the infrared-corrected coordinates to obtain the coordinates of the four corner points of the target bounding box in the original infrared image (with distortion). The inverse distortion calculation includes: converting the infrared-corrected coordinates into normalized coordinates, calculating the radial distortion factor and tangential distortion compensation based on the infrared distortion coefficient, and then obtaining the distorted coordinates of the original infrared image.

[0041] (5) Generate infrared tags Based on the coordinates of the four corner points of the target bounding box in the original infrared image, calculate the center coordinates, width, and height of the target, and normalize them to the infrared image size. This generates infrared tags that conform to target detection formats (such as YOLO).

[0042] The present invention will be further illustrated below through examples: This embodiment uses visible light images (3840×2160) and infrared images (640×512) from a dual-channel camera as examples to illustrate the implementation process of the present invention in detail.

[0043] Step 1: Obtain basic data and parameters 1) Visible light labels: using YOLO format, based on 3840×2160 image annotations, including target category and normalized center coordinates. Width and height .

[0044] 2) Visible light camera parameters: a. Intrinsic parameter matrix : [2874.51331901376, -39.3039161410885, 1996.59506783266; 0,2797.45100605375, 1527.01984644443; 0,0,1] b. Distortion coefficient :[k1=0.3213,k2=-1.4223 p1=0.0092,p2=0.0025, k3=2.0785] c. Visible light original image size: 4000×3000.

[0045] 3) Infrared camera parameters: a. Intrinsic parameter matrix : [750.182109454864, -0.0588692810727856, 345.630330701034; 0,744.088469963417,290.874367931254; 0,0,1] b. Distortion coefficient : [k1=-0.3459, k2=-0.0695, p1=0.0011, p2=-0.0090, k3=0.3828] c. Raw infrared image size: 640×512.

[0046] 4) Overlapping area: the upper left corner (615, 419) and the lower right corner (3377, 2557) in the corrected visible light image.

[0047] Step 2: Visible light tag coordinate transformation and correction Normalized coordinates of visible light YOLO tags Convert the pixel coordinates to the original visible light image (3840×2160) and calculate the four corner points of the target bounding box (top left, top right, bottom left, bottom right).

[0048] The visible light image is enlarged to 4000×3000, and the enlargement offset is calculated. =80, =420), mapping the coordinates of the four corner points to the expanded image.

[0049] Call the standard distortion correction function (such as cv2.undistortPoints) and utilize the visible light intrinsic parameters. and distortion coefficient Correct the expanded coordinates, such as Figure 2 As shown in (a) and (b), the target coordinates in the corrected visible light image are obtained.

[0050] Step 3: Infrared Image Correction Call the standard distortion correction function (such as cv2.undistortPoints) and utilize the infrared intrinsic parameters. and distortion coefficient Correct the coordinates, such as Figure 3 As shown in (a) and (b), the target coordinates in the corrected infrared image are obtained.

[0051] Step 4: Coordinate mapping of overlapping areas The corrected visible light target coordinates are cropped to the overlapping region (615≤). ≤3377,419≤ ≤2557), overlapping areas such as Figure 4 As shown, coordinates outside the region are taken as boundary values.

[0052] Calculate the size ratio of the overlapping area to the infrared-corrected image (width ratio = 640 / (3377-615), height ratio = 512 / (2557-419)), and map the cropped coordinates to the infrared-corrected coordinates.

[0053] Step 5: Infrared coordinate inverse distortion transformation Inverse distortion calculation is performed on the infrared-corrected coordinates: 1) Adjust the coordinates after infrared correction Convert to normalized coordinates : , (in , , , Infrared internal reference (parameters).

[0054] 2) Calculate the radial distortion factor based on the distortion coefficient of the infrared camera:

[0055] In the formula, The radial distortion coefficient of the infrared camera is denoted as . It is the sum of squares of the normalized coordinates.

[0056] 3) Calculate tangential distortion compensation based on the distortion coefficient of the infrared camera:

[0057] In the formula, This represents the tangential distortion coefficient of the infrared camera. , for Directional compensation amount Directional compensation amount.

[0058] 4) Based on the radial distortion factor and the tangential distortion compensation, the normalized distortion coordinates of the original infrared image are obtained:

[0059] 5) Convert the normalized distortion coordinates to pixel coordinates of the original infrared image:

[0060] Step 6: Generate infrared tags Calculate the center coordinates based on the coordinates of the four corner points of the target bounding box in the original infrared image. ,Width( ),high( The data is normalized to a size of 640×512 and used to generate an infrared tag in YOLO format.

[0061] Based on the infrared channel tag generation method for dual-channel cameras proposed above, a computer program is formed, comprising: 1) Define the infrared coordinate inverse distortion function inverse_distort_points to implement the inverse distortion calculation logic in step 4 above.

[0062] 2) Define the main function `convert_visible_to_infrared_label` for label transformation, and execute the coordinate transformation and label generation process from step 2 to step 5 in sequence: Read the visible light label file and parse the target category and normalized coordinates; Complete the expansion of visible light coordinates, distortion correction, and overlapping region clipping; Mapped to infrared-corrected coordinates and subjected to inverse distortion transformation; Calculate the target bounding box parameters in the original infrared image, and generate and save the infrared label file.

[0063] 3) Configure the dual-channel camera parameters (intrinsic parameters, distortion coefficients, overlapping areas, etc.) and call the main function to implement label conversion.

[0064] The embodiments described above are merely preferred embodiments of the present invention. Ordinary variations and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating infrared channel tags applicable to dual-channel cameras, characterized in that, include: Acquire the visible light target detection tags acquired by the dual-channel camera's visible light camera, the intrinsic parameters and distortion coefficients of the visible light camera and the infrared camera, the original image size, and the coordinate range of the overlapping area of ​​the visible light and infrared fields of view; The normalized coordinates of the visible light target detection label are converted into pixel coordinates of the visible light image to obtain the coordinates of the four corner points of the target bounding box; based on the intrinsic parameter matrix and distortion coefficients of the visible light camera, distortion correction is performed on the visible light image to obtain the corrected visible light image and the coordinates of the four corner points of the target bounding box in the image. Based on the intrinsic parameter matrix and distortion coefficients of the infrared camera, distortion correction is performed on the infrared image to obtain the corrected infrared image. Based on the coordinate range of the overlapping area between the visible light and infrared fields of view, the area overlapping with the infrared field of view is cropped from the corrected visible light image, and the coordinates of the four corner points of the target bounding box are restricted to the overlapping area; based on the size ratio of the overlapping area and the infrared corrected image, the target coordinates in the overlapping area are mapped to the infrared corrected coordinate system. Using the intrinsic parameter matrix and distortion coefficients of the infrared camera, inverse distortion calculation is performed on the infrared-corrected coordinates to obtain the coordinates of the four corner points of the target bounding box in the original infrared image, and then an infrared target detection label conforming to the target detection format is generated.

2. The method for generating infrared channel tags for a dual-channel camera according to claim 1, characterized in that, Inverse distortion calculation is performed on the infrared-corrected coordinates: 1) Adjust the coordinates after infrared correction Convert to normalized coordinates : ， In the formula, , , , These are the parameters of the infrared camera's intrinsic parameter matrix; 2) Calculate the radial distortion factor and tangential distortion compensation based on the distortion coefficient of the infrared camera; In the formula, The radial distortion coefficient of the infrared camera is denoted as . The sum of squares of the normalized coordinates. Radial distortion factor; In the formula, This represents the tangential distortion coefficient of the infrared camera. , For the distortion coefficient Directional compensation amount Directional compensation amount; 3) Based on the radial distortion factor and the tangential distortion compensation amount, the normalized distortion coordinates of the original infrared image are obtained, and then converted into the pixel coordinates of the original infrared image.

3. The method for generating infrared channel tags for a dual-channel camera according to claim 2, characterized in that, Based on the radial distortion factor and the tangential distortion compensation, the normalized distortion coordinates of the original infrared image are obtained. : Convert the normalized distortion coordinates to pixel coordinates of the original infrared image: In the formula, These are the pixel coordinates of the original infrared image.

4. The method for generating infrared channel tags for a dual-channel camera according to claim 1, characterized in that, Based on the coordinates of the four corner points of the target bounding box in the original infrared image, infrared target detection labels conforming to the target detection format are generated, where: Calculate the center coordinates: Calculate the width: Calculate height: In the formula, the coordinates of the four corner points of the target bounding box in the original infrared image are respectively , , , ; Based on the center coordinates, width, and height, the data is normalized to the original infrared image size to generate infrared target detection tags in YOLO format.

5. The method for generating infrared channel tags for a dual-channel camera according to claim 1, characterized in that, If the visible light image acquired by the visible light camera is not a full-frame image, the visible light image is enlarged to the original image size, and the coordinates of the four corner points of the target bounding box are mapped into the enlarged image. Then, based on the intrinsic parameter matrix and distortion coefficients of the visible light camera, distortion correction is performed on the enlarged image to obtain the corrected visible light image and the coordinates of the four corner points of the target bounding box in the image.

6. The method for generating infrared channel tags for a dual-channel camera according to claim 1, characterized in that, If the infrared image captured by the infrared camera is not a full-frame image, the infrared image is enlarged to the original image size, and then the distortion is corrected on the enlarged image based on the intrinsic parameter matrix and distortion coefficient of the infrared camera to obtain the corrected infrared image.

7. The method for generating infrared channel tags for a dual-channel camera according to claim 1, characterized in that, Based on the size ratio of the overlapping region to the infrared-corrected image, the target coordinates within the overlapping region are mapped to the infrared-corrected coordinate system, specifically as follows: The coordinates of the visible light target confined within the overlapping region are mapped to the coordinate system of the infrared-corrected image. During the mapping, the overlapping region of the visible light-corrected image and the infrared-corrected image correspond completely in space, only differing in size. Let the coordinates of the upper left corner of the overlapping region in the corrected visible light image be... The coordinates of the lower right corner are The width of the overlapping region is The height is ; For any point within the overlapping region of the visible light corrected image The corresponding coordinates in the distortion-free coordinate system of the infrared-corrected image for: ， In the formula, the width of the image after infrared correction is Height is .

8. The method for generating infrared channel tags for a dual-channel camera according to claim 1, characterized in that, Intrinsic parameter matrix based on visible light camera and distortion coefficient The distortion correction function is called to correct the distortion of the four corner points of the target bounding box in the expanded image.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

Images