A zoomable multi-channel camera fusion device and method
By disassemblying the multi-channel camera fusion into image fusion between two cameras, combining image mapping relationship and gimbal position, the problem of spatial position distinction and label synchronization in telephoto mode is solved, and the multi-channel camera is realized Accurate image matching and label synchronization between them improve the accuracy of the monitoring system and multi-environment adaptability.
Patent Information
- Application Number
- CN202411179436.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-27
AI Technical Summary
Existing multi-channel cameras have difficulty distinguishing spatial positions in telephoto mode and cannot synchronize label positions between different cameras, resulting in poor image fusion and difficult real-time response.
By disassembling the multiple camera fusion into image fusion between two cameras, combining the image mapping relationship between cameras and the position relationship between the cameras, the image mapping matrix is calculated, and image feature matching and label synchronization between cameras in different imaging principles are achieved. Overcome image distortion problem in small field telephoto mode.
It realizes accurate image matching and positioning between multiple cameras, improves the accuracy and convenience of the monitoring system, enhances the adaptability and flexibility of multiple environments, and solves the problem of difficulty in distinguishing position under the long focal length.
Smart Images

Figure CN118741309B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of optical security monitoring, and in particular to a device and method for fusion of multiple cameras with variable focus. Background Art
[0002] At present, the commonly used multi-channel cameras in the security monitoring field are composed of color cameras, thermal imagers and infrared fill light modules, but this combination only achieves the purpose of multi-channel acquisition and cannot effectively integrate the data of multiple cameras. Especially in telephoto mode, due to the small field of view, especially for thermal imagers, it is difficult to distinguish the specific spatial position of the current viewing area.
[0003] The traditional multi-channel fusion method uses image feature matching, but there are the following problems. First, the image feature calculation and analysis is complex and highly dependent on the device computing resources. It is often difficult for lightweight front-end devices to meet such computing requirements. Second, due to the differences in imaging principles, it is difficult for multi-channel cameras to perform graphic feature calculations and complete image matching. Third, due to changes in the horizontal angle, tilt angle, and focal length of the equipment during operation, at higher tilt angles, the area of interest appears to move on the screen in an arc trajectory, resulting in image distortion. Traditional image matching algorithms are difficult to achieve fusion purposes, and it is difficult to meet the requirements of real-time response in monitoring systems. It is time-consuming and labor-intensive, and the effect cannot be guaranteed.
[0004] In addition, similar PZT surveillance cameras currently on the market can bind labels to targets, but only for single-channel cameras. It is impossible to mark on one camera and synchronously display the label position on another camera at the same time. In view of this, the present invention discloses a multi-channel camera fusion device and method with variable focus, which aims to solve the problem that the existing surveillance cameras cannot locate the position of the viewing area when performing panning, tilting operations, and long focal length operation, and solve the problem that the label position cannot be shared during the operation of multiple cameras. Summary of the invention
[0005] In view of the above problems, a zoomable multi-channel camera fusion device and method are provided. By decomposing the multi-channel camera fusion into image fusion between two cameras, the image mapping relationship at the same focal length between cameras and the mapping of the relationship between the surround view image and the pan-tilt position are combined to realize image feature calculation and image matching for specific spatial positions between multi-channel cameras with different imaging principles, thereby overcoming the distinction and matching positioning of specific spatial positions and camera imaging areas in the small field of view telephoto mode; by synchronizing the label positions between different multi-channel cameras, synchronous label binding and position display of the multi-channel cameras are realized; by calculating the image mapping matrix between different types of cameras, accurate correspondence and fusion between images collected by each camera are realized, thereby improving the quality of image fusion; by mutual conversion between image coordinates and absolute angular coordinates between two cameras, accurate matching and positioning of image positions between cameras are realized, the algorithm is simple and the image matching and positioning effect is good, thereby solving the image distortion problem caused by changes in device angles and arc movement of interest areas between camera images, thereby improving the accuracy of image fusion.
[0006] To achieve the above object, the technical solution adopted by the present invention is as follows: a method for fusion of multi-channel cameras with variable focus, comprising a multi-channel camera and a two-axis gimbal, wherein the multi-channel camera comprises at least two cameras of different types, and the image fusion between the multi-channel cameras can be decomposed into the mutual fusion of images between two cameras, wherein the image fusion of two different types of cameras comprises the following steps:
[0007] Step 1: Get the center position of the optical axis of camera 1 and camera 2 (x 1center ,y 1center ), (x 2center ,y 2center ), and obtain the detector physical sizes of camera 1 and camera 2 (h1, w1), (h2, w2);
[0008] Step 2: Calculate the image mapping matrices M1 and M2 of camera 1 and camera 2 at the same focal length f0;
[0009] Step 3: Adjust the focal length of the camera lens to the intermediate focal length f mid , based on the two-axis pan-tilt rotation, establish a surround image Src , and establish a corresponding relationship table M between the surround view image and the two-axis gimbal position;
[0010] Step 4: Select the region of interest in the image interface of camera 2. The image coordinates of the four vertices of the region of interest are A(x r1 ,y r1 )、B(x r2 ,y r2 )、C(x r3 ,y r3 )、D(x r4 ,yr4 ), the four vertices of the region of interest surround the image Src 0;
[0011] Step 5: According to the image mapping matrix M2 in step 2, obtain the image Src The image coordinates of the four vertices in 0 corresponding to camera 1 are A'(x1, y1), B'(x2, y2), C'(x3, y3), and D'(x4, y4);
[0012] Step 6: Convert the image coordinates corresponding to camera 1 in step 5 to the absolute angle coordinates corresponding to camera 2
[0013] A'(x1,y1)→A''(Θ1,Φ1)
[0014] B'(x2,y2)→B''(Θ2,Φ2)
[0015] C'(x3,y3)→C''(Θ3,Φ3)
[0016] D'(x4,y4)→D''(Θ4,Φ4)
[0017] Step 7: Convert the absolute angle coordinates of camera 2 described in step 6 to the image coordinates of camera 1
[0018] A''(Θ1,Φ1)→a'( X1C , Y 1C )
[0019] B''(Θ2,Φ2)→b'(X 2C , Y 2C )
[0020] C''(Θ3,Φ3)→c'(X 3C , Y 3C )
[0021] D''(Θ4,Φ4)→d'(X 4C , Y 4C )
[0022] Step 8: According to the current two-axis gimbal angle θy, Φy, obtain the surround image of camera 1 Src The image slice in the camera image is extracted to obtain the image coordinates a'(X 1C , Y 1C ), b'(X 2C , Y 2C ), c'(X 3C , Y 3C ), d'(X 4C , Y 4C ) and enclose the image area Src1. Realize the transmission and positioning of the corresponding tag positions in camera 1 and camera 2;
[0023] Step 9: The image fusion between camera 1 and camera 2 is finally displayed as
[0024] Src out = a × Src 0 + (1- a ) × Src 1
[0025] in a is the fusion coefficient, and its value range is [0,1].
[0026] Preferably, the camera 1 and the camera 2 can be respectively one of a thermal imaging camera, a white camera, a color camera, a black and white camera and an infrared camera.
[0027] Preferably, the image mapping matrices M1 and M2 in step 2 satisfy the following conditions:
[0028]
[0029] in,( x , y ) are the coordinates in the original image; ( x ', y ') are the coordinates of the object in the image; A, B, C, D are the matrix elements that control rotation, scaling, and cropping; T x , T y is the translation amount.
[0030] Preferably, the horizontal angle resolution of the corresponding relationship table M between the surround view image and the two-axis gimbal position in step 3 satisfies fovx≤2arctan(w1 / (2f max )); vertical angle resolution satisfies fovy≤2arctan(h1 / (2f max )), where f max is the longest focal length of the camera; w1 is the width of the camera-detector; h1 is the height of the camera-detector.
[0031] Preferably, the conversion formula for converting the image coordinates corresponding to camera 1 into the absolute angle coordinates corresponding to camera 2 in step 6 is as follows:
[0032]
[0033] Where Θy and Φy are the current horizontal angle and pitch angle of the two-axis gimbal respectively;
[0034] γ satisfy ;
[0035] α , β satisfy ;
[0036] Xfovr is the horizontal field of view of camera 2, satisfying
[0037] Yfovr is the vertical field of view of camera 2, satisfying
[0038] Among them, Xn and Yn are the pixel positions of the point in the second camera image; X pixels , Y pixels is the number of pixels of the width and height of the camera 2 image; fr is the current focal length of the camera 2 lens; w2 is the width of the camera 2 detector; h2 is the height of the camera 2 detector.
[0039] Preferably, the conversion formula for converting the absolute angle coordinates in the corresponding camera 2 to the image coordinates under the camera 1 image in step 7 is:
[0040]
[0041]
[0042]
[0043] Where Xfovc is the horizontal field of view of the camera; Yfovc is the vertical field of view of the camera; h1 is the height of the camera detector; w1 is the width of the camera detector; X pixels , Y pixels is the number of pixels of width and height of the second camera image; f mid The focal length of the camera lens when the focal length is adjusted to the middle;
[0044] δ satisfy ;
[0045] ε satisfy ; Wherein, Θc is the horizontal angle change of the two-axis gimbal.
[0046] Preferably, during the conversion of the absolute angular coordinates into the image coordinates under the camera image, the horizontal angle and the pitch angle of the two-axis gimbal remain unchanged, and the focal length is the focal length f when the focal length of the camera lens is adjusted to the middle in step 3. mid .
[0047] A zoomable multi-channel camera fusion device comprises a color camera; a thermal imaging camera; an infrared fill light laser used for infrared fill light to assist the color camera and the thermal imaging camera in clear imaging; an infrared receiving camera used to receive the reflected light of the infrared fill light laser on a target object and to perform imaging and output; a laser gimbal used to adjust the horizontal and pitch angle rotation of the infrared fill light laser; a two-axis gimbal used to adjust the pointing position of the entire multi-channel camera fusion device and to send back position information in real time, and a main control module used to control the rotation of each module and collect and send back data; the color camera, thermal imaging camera, infrared fill light laser, infrared receiving camera, laser gimbal and main control module are mounted and fixed on the two-axis gimbal.
[0048] Preferably, the color camera and thermal imaging camera are 40x continuous zoom cameras that can transmit focal length data back.
[0049] Preferably, the irradiation position and laser spot size of the infrared fill light laser can be automatically adjusted according to the focal length data of the infrared receiving camera to ensure that the imaging of the irradiation position is located in the middle of the infrared receiving camera.
[0050] Due to the adoption of the above technical solution, the present invention has the following beneficial effects.
[0051] (1) The present invention decomposes the fusion of multiple cameras into image fusion between two cameras, combines the image mapping relationship between cameras at the same focal length, and the mapping of the relationship between the surround view image and the pan / tilt position, to achieve image feature calculation and image matching for specific spatial positions between multiple cameras with different imaging principles, overcomes the distinction and matching positioning between specific spatial positions and camera imaging areas in the small field of view telephoto mode, and achieves precise positioning, rapid matching and image fusion of images of the interest area at the same spatial position between cameras, thereby enhancing the accuracy of monitoring.
[0052] (2) The present invention synchronizes the label positions between different cameras, thereby achieving synchronized label binding and position display for multiple cameras even at different focal lengths, thereby achieving consistent monitoring views, label position transmission and positioning; by calculating the image mapping matrix between different types of cameras, the precise correspondence and fusion between the images collected by each camera is achieved, the quality of image fusion is improved, and thus the accuracy and convenience of monitoring are improved.
[0053] (3) The present invention achieves accurate matching and positioning of image positions between cameras by converting the image coordinates of camera two into the image coordinates of camera one, the absolute angular coordinates of camera two, and finally converting them back into the image coordinates of camera one. The algorithm is simple and the image matching and positioning effect is good. It solves the image distortion problem caused by the change of device angle and the arc movement of the area of interest between camera images, and improves the accuracy of image fusion.
[0054] (4) The present invention proposes a method for fusing multiple camera images based on spatial position, which can achieve effective imaging in a completely dark environment. At the same time, based on the spatial position and image fusion method, it can enrich the image information in scenes with insufficient features such as thermal imaging and infrared fill light cameras, and at the same time realize the function of multi-camera position indication and label transmission in PTZ mode.
[0055] (5) The present invention uses multiple zoom cameras, including color cameras, thermal imaging cameras, etc., so that the device can capture images in different bands, thereby enhancing the multi-environment adaptability and flexibility of the monitoring system. By supporting panoramic imaging and indicating and fusing the imaging positions of other cameras, the problem of difficulty in distinguishing the orientation under long focal length is solved, and the orientation perception capability of the monitoring is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The following is a detailed discussion of the making and application of the preferred embodiments of the present invention. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in various specific environments. The specific embodiments discussed are only for the purpose of illustrating the specific ways of making and using the present invention, and do not limit the scope of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.
[0057] Figure 1 It is a schematic diagram of the process of the multi-channel camera fusion method of the present invention.
[0058] Figure 2 It is a schematic diagram of the conversion process of the interest areas of camera one and camera two of the present invention.
[0059] Figure 3 Schematic diagram of the change of point A in the image at different two-axis gimbal positions of the present invention.
[0060] Figure 4 Schematic diagram of a variable-focus multi-camera device according to the present invention.
[0061] Figure 5 Schematic diagram of cross-camera annotation and image fusion of the present invention.
[0062] Among them, 1-color camera; 2-thermal imaging camera; 3-infrared receiving camera; 4-infrared fill light laser; 5-laser gimbal; 6-two-axis gimbal; 7-main control module; a-schematic diagram of the vertices in the area of interest of camera two; b-schematic diagram of the vertices converted to the same focal length in camera one; c-schematic diagram of the absolute angular coordinates in camera two; d-schematic diagram of the image vertex positions at the target focal length in camera one; e-schematic diagram of the positions synchronized to the visible light image; f-schematic diagram of manual position marking in thermal images; g-schematic diagram of image fusion. DETAILED DESCRIPTION
[0063] The following is a detailed discussion of the making and application of the preferred embodiments of the present invention. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in various specific environments. The specific embodiments discussed are only intended to illustrate specific ways to make and use the present invention and do not limit the scope of the present invention.
[0064] Step 1: Get the center position of the optical axis of camera 1 and camera 2 (x 1center ,y 1center ), (x 2center ,y 2center ), and obtain the detector physical sizes (h1, w1), (h2, w2) of camera 1 and camera 2.
[0065] Step 2: Calculate the image mapping matrices M1 and M2 of camera 1 and camera 2 at the same focal length f0.
[0066] The image mapping matrices M1 and M2 in step 2 satisfy the following conditions
[0067]
[0068] in,( x , y ) are the coordinates in the original image; ( x ', y ') are the coordinates of the object in the image; A, B, C, D are the matrix elements that control rotation, scaling, and cropping; T x , T y is the translation amount.
[0069] Since the transformation equations have 6 unknowns, we need to find 3 sets of mapping points to solve them. The three points just determine a plane. In the two camera images, we select 3 sets of mapping points A, B, C, D, and T. x , T y Parameter calculation.
[0070] Step 3: Adjust the focal length of the camera lens to the intermediate focal length f mid , based on the two-axis pan / tilt 6 rotation, establish a surround image Src , and establish a corresponding relationship table M between the surround view image and the position of the two-axis gimbal 6. The horizontal angle resolution of the corresponding relationship table M satisfies fovx≤2arctan(w1 / (2f max )); vertical angle resolution satisfies fovy≤2arctan(h1 / (2f max )), where f max is the longest focal length of the camera; w1 is the width of the camera-detector; h1 is the height of the camera-detector.
[0071] Step 4: Figure 2-a is a schematic diagram of the vertices of the interest area of camera 2. The interest area is selected in the image interface of camera 2. The image coordinates of the four vertices of the interest area are A(x r1 ,y r1 )、B(x r2 ,y r2 )、C(x r3 ,y r3 )、D(x r4 ,y r4 ), the four vertices of the region of interest surround the image Src 0.
[0072] Step 5: Figure 2 -b is a schematic diagram of the conversion to the vertex diagram of the same focal length in camera 1. According to the image mapping matrix M2 in step 2, the image is obtained Src The image coordinates of the four vertices in 0 in camera 1 are A'(x1, y1), B'(x2, y2), C'(x3, y3), and D'(x4, y4).
[0073] Step 6: Figure 2 -c shows the absolute angle coordinates of camera 2. Convert the image coordinates corresponding to camera 1 in step 5 to the absolute angle coordinates of camera 2.
[0074] A'(x1,y1)→A''(Θ1,Φ1)
[0075] B'(x2,y2)→B''(Θ2,Φ2)
[0076] C'(x3,y3)→C''(Θ3,Φ3)
[0077] D'(x4,y4)→D''(Θ4,Φ4)
[0078] The conversion formula for converting the image coordinates corresponding to the camera 1 to the absolute angle coordinates corresponding to the camera 2 is as follows:
[0079]
[0080] Where Θy and Φy are the current horizontal angle and pitch angle of the two-axis gimbal 6 respectively;
[0081] γ satisfy ;
[0082] α , β satisfy ;
[0083] Xfovr is the horizontal field of view of camera 2, satisfying ;
[0084] Yfovr is the vertical field of view of camera 2, satisfying ;
[0085] Among them, Xn and Yn are the pixel positions of the point in the second camera image; X pixels , Y pixels is the number of pixels of the width and height of the camera 2 image; fr is the current focal length of the camera 2 lens; w2 is the width of the camera 2 detector; h2 is the height of the camera 2 detector.
[0086] Step 7: Figure 2 -d shows the schematic diagram of the image vertex position under the target focal length of camera one. Convert the absolute angle coordinates in camera two described in step 6 to the image coordinates under the image of camera one
[0087] A''(Θ1,Φ1)→a'( X1C , Y 1C )
[0088] B''(Θ2,Φ2)→b'(X 2C , Y 2C )
[0089] C''(Θ3,Φ3)→c'(X 3C , Y 3C )
[0090] D''(Θ4,Φ4)→d'(X 4C , Y 4C )
[0091] The conversion method between the coordinates is the reverse reasoning of step 6. During the conversion of the absolute angle coordinates to the image coordinates under the camera image, the horizontal angle and the pitch angle of the two-axis gimbal 6 remain unchanged, and the focal length is the focal length f when the focal length of the camera lens is adjusted to the middle in step 3. mid At this time, the conversion formula for converting the absolute angle coordinates in the corresponding camera 2 to the image coordinates under the camera 1 image is:
[0092]
[0093]
[0094]
[0095] Where Xfovc is the horizontal field of view of the camera; Yfovc is the vertical field of view of the camera; h1 is the height of the camera detector; w1 is the width of the camera detector; X pixels , Y pixels is the number of pixels of width and height of the second camera image; f midThe focal length of the camera lens when the focal length is adjusted to the middle;
[0096] δ satisfy ;
[0097] ε satisfy ; Wherein, Θc is the horizontal angle change of the two-axis gimbal 6.
[0098] Step 8: According to the current two-axis gimbal 6 angles θy, Φy, obtain the surround view image of camera 1 Src The image slice in the camera image is extracted to obtain the image coordinates a'(X 1C , Y 1C ), b'(X 2C , Y 2C ), c'(X 3C , Y 3C ), d'(X 4C , Y 4C ) and enclose the image area Src 1. Realize the transfer and positioning of the corresponding tag positions in camera 1 and camera 2.
[0099] Step 9: The image fusion between camera 1 and camera 2 is finally displayed as Src out = a × Src 0 + (1- a )× Src 1, where a is the fusion coefficient, and its value range is [0,1].
[0100] A multi-channel camera fusion device with variable focus comprises a color camera 1; a thermal imaging camera 2; an infrared fill light laser 4 used for infrared fill light to assist the color camera 1 and the thermal imaging camera 2 in clear imaging; an infrared receiving camera 3 used to receive the reflected light of the infrared fill light laser 4 on a target object and to perform imaging and output; a laser gimbal 5 used to adjust the horizontal and pitch angle rotation of the infrared fill light laser 4; a two-axis gimbal 6 used to adjust the pointing position of the entire multi-channel camera fusion device and to send back position information in real time, and a main control module 7 used to control the rotation of each module, collect and send back data; the color camera 1, the thermal imaging camera 2, the infrared fill light laser 4, the infrared receiving camera 3, the laser gimbal 5 and the main control module 7 are installed and fixed on the two-axis gimbal 6.
[0101] The color camera 1 and thermal imaging camera 2 are 40x continuous zoom cameras that can transmit focal length data. The irradiation position and laser spot size of the infrared fill light laser 4 can be adjusted automatically according to the focal length data of the infrared receiving camera 3 to ensure that the image at the irradiation position is located in the middle of the infrared receiving camera 3.
[0102] The following is combined with Figure 1-3 Provide a detailed description. Example 1
[0103] The present invention proposes a method for fusing multiple camera images based on spatial position, which can realize effective imaging in a completely dark environment. At the same time, based on the spatial position and image fusion method, it can enrich the image information in scenes with insufficient features such as thermal imaging and infrared fill light cameras, and realize the function of multi-camera position indication and label transmission in PTZ mode.
[0104] like Figure 1 A multi-channel camera fusion method with variable focus is shown, comprising the following steps.
[0105] Step S1: Obtain the intrinsic parameters and physical dimensions of each camera.
[0106] Step S2: Obtain the image mapping relationship between each camera at the same focal length f0.
[0107] Step S3: Establishing a mapping between the surround view image and the PTZ position.
[0108] Step S4: Obtain the image coordinates, focal length and gimbal angle of the current camera's area of interest.
[0109] Step S5: Camera image coordinate conversion.
[0110] Step S6: Convert image coordinates into absolute angular coordinates.
[0111] Step S7: convert the diagonal coordinates into image coordinates under the current focal length and gimbal angle.
[0112] Step S8: reference image capture, obtain image slices of the surround image, and complete the tag position transfer and positioning between each camera.
[0113] Step S9: image fusion is completed.
[0114] This embodiment takes the thermal imaging camera 2 and the color camera 1 as examples. Figure 4 When the position of the two-axis gimbal 6 of the device is changed from (Θ, Φ) to (Θ-Θ1, Φ+Φ1), the specific implementation method of the image position A transformation and image fusion at different focal lengths is as follows: Figure 1-3 As shown, the specific steps of a multi-channel camera fusion method with variable focus are as follows.
[0115] Step 1: Get the center position of the optical axis of color camera 1 and thermal imaging camera 2 (x 1center ,y 1center ), (x 2center ,y 2center), and obtain the detector physical sizes (h1, w1), (h2, w2) of color camera 1 and thermal imaging camera 2.
[0116] Step 2: Calculate the image mapping matrices M1 and M2 of the color camera 1 and the thermal imaging camera 2 at the same focal length f0.
[0117] The image mapping matrices M1 and M2 in step 2 satisfy the following conditions
[0118]
[0119] in,( x , y ) are the coordinates in the original image; ( x ', y ') are the coordinates of the object in the image; A, B, C, D are the matrix elements that control rotation, scaling, and cropping; T x , T y is the translation amount.
[0120] Since the transformation equations have 6 unknowns, we need to find 3 sets of mapping points to solve them. The three points just determine a plane. In the two camera images, we select 3 sets of mapping points A, B, C, D, and T. x , T y Parameter calculation.
[0121] Step 3: Adjust the focal length of the color camera 1 lens to the middle focal length f mid , based on the two-axis pan / tilt 6 rotation, establish a surround image Src , and establish a corresponding relationship table M between the surround view image and the position of the two-axis gimbal 6. The horizontal angle resolution of the corresponding relationship table M satisfies fovx≤2arctan(w1 / (2f max )); vertical angle resolution satisfies fovy≤2arctan(h1 / (2f max )), where f max is the longest focal length of the camera; w1 is the width of the color camera 1 detector; h1 is the height of the color camera 1 detector.
[0122] Step 4: Figure 2 -a is a schematic diagram of the vertices of the interest area of the thermal imaging camera 2. The interest area is selected in the image interface of the thermal imaging camera 2. The image coordinates of the four vertices of the interest area are A(x r1 ,y r1 )、B(x r2 ,y r2 )、C(x r3 ,y r3 )、D(x r4 ,y r4), the four vertices of the region of interest surround the image Src 0.
[0123] Step 5: Figure 2 -b is a schematic diagram of the conversion to the vertex diagram of the same focal length in the color camera 1. According to the image mapping matrix M2 in step 2, the image is obtained Src The image coordinates of the four vertices in 0 in color camera 1 are A'(x1, y1), B'(x2, y2), C'(x3, y3), and D'(x4, y4).
[0124] Step 6: Figure 2 -c shows the absolute angle coordinates of thermal imaging camera 2. The image coordinates corresponding to color camera 1 in step 5 are converted into the absolute angle coordinates of thermal imaging camera 2.
[0125] A'(x1,y1)→A''(Θ1,Φ1)
[0126] B'(x2,y2)→B''(Θ2,Φ2)
[0127] C'(x3,y3)→C''(Θ3,Φ3)
[0128] D'(x4,y4)→D''(Θ4,Φ4)
[0129] The conversion formula for converting the image coordinates corresponding to the color camera 1 into the absolute angular coordinates in the corresponding color camera 1 is as follows:
[0130]
[0131] Where Θy and Φy are the current horizontal angle and pitch angle of the two-axis gimbal 6 respectively;
[0132] γ satisfy ;
[0133] α , β satisfy ;
[0134] Xfovr is the horizontal field of view of thermal imaging camera 2, satisfying ;
[0135] Yfovr is the vertical field of view of thermal imaging camera 2, satisfying ;
[0136] Among them, Xn and Yn are the pixel positions of the point in the image of thermal imaging camera 2; X pixels , Y pixelsis the number of pixels of the width and height of the thermal image camera 2 image; fr is the current focal length of the thermal image camera 2 lens; w2 is the width of the thermal image camera 2 detector; h2 is the height of the thermal image camera 2 detector.
[0137] Step 7: Figure 2 The absolute angle coordinates in the corresponding thermal imaging camera 2 described in step 6 are converted into the image coordinates under the image of the color camera 1.
[0138] A''(Θ1,Φ1)→a'( X1C , Y 1C )
[0139] B''(Θ2,Φ2)→b'(X 2C , Y 2C )
[0140] C''(Θ3,Φ3)→c'(X 3C , Y 3C )
[0141] D''(Θ4,Φ4)→d'(X 4C , Y 4C )
[0142] The conversion method between the coordinates is the reverse reasoning of step 6. During the conversion of the absolute angle coordinates into the image coordinates under the image of the color camera 1, the horizontal angle and the pitch angle of the two-axis gimbal 6 remain unchanged, and the focal length is the focal length f when the focal length of the lens of the color camera 1 is adjusted to the middle in step 3. mid At this time, the conversion formula for converting the absolute angular coordinates in the corresponding thermal imaging camera 2 into the image coordinates under the image of the color camera 1 is:
[0143]
[0144]
[0145]
[0146] Where Xfovc is the horizontal field of view of color camera 1; Yfovc is the vertical field of view of color camera 1; h1 is the height of the detector of color camera 1; w1 is the width of the detector of color camera 1; X pixels , Y pixels f is the number of pixels of the width and height of the image of thermal camera 2; mid The focal length of the color camera 1 when the lens focal length is adjusted to the middle;
[0147] δ satisfy ;
[0148] ε satisfy ; Wherein, Θc is the horizontal angle change of the two-axis gimbal 6.
[0149] Step 8: According to the current two-axis gimbal 6 angles Θy, Φy, obtain the surround image of the color camera 1 Src The image slice in the color camera 1 image is extracted with the image coordinates a'(X 1C , Y 1C ), b'(X 2C , Y 2C ), c'(X 3C , Y 3C ), d'(X 4C , Y 4C ) and enclose the image area Src 1. Realize the transmission and positioning of the corresponding label positions in the color camera 1 and the thermal imaging camera 2.
[0150] Step 9: The image fusion between color camera 1 and thermal camera 2 is finally displayed as Src out = a × Src 0 +(1- a ) × Src 1, where a is the fusion coefficient, and its value range is [0,1]. Figure 5 As shown, the final image fusion diagram is as follows Figure 5 -g. The schematic diagram of manual position marking in thermal imaging camera 2 is shown in Figure 5 -f, which is synchronized to the position of the visible light image of color camera 1 as shown in Figure 5 -e as shown.
[0151] The present invention decomposes multi-channel camera fusion into image fusion between two cameras, combines the image mapping relationship at the same focal length between cameras, and the mapping of the relationship between the surround view image and the pan / tilt position, so as to realize the image feature calculation and image matching for a specific spatial position between multi-channel cameras with different imaging principles; synchronizes the label positions between different multi-channel cameras to realize synchronized label binding and position display of the multi-channel cameras; calculates the image mapping matrix between different types of cameras to realize the precise correspondence and fusion between the images collected by each camera, and improves the quality of image fusion; and realizes the precise matching and positioning of the image positions between cameras through the mutual conversion between the image coordinates and the absolute angular coordinates between two cameras, and solves the image distortion problem caused by the change of the device angle and the arc movement of the interest area between the camera images. Example 2
[0152] This embodiment uses multiple zoom cameras, including color camera 1, thermal imaging camera 2, etc., so that the device can capture images of different bands, thereby enhancing the multi-environment adaptability and flexibility of the monitoring system. By supporting panoramic imaging and indicating and fusing the imaging positions of other cameras, the problem of difficulty in distinguishing directions under long focal lengths is solved, and the direction perception capability of monitoring is enhanced.
[0153] like Figure 4 The multi-channel camera fusion device with variable focus shown in the figure includes a color camera 1; a thermal imaging camera 2; an infrared fill light laser 4 used for infrared fill light to assist the color camera 1 and the thermal imaging camera 2 in clear imaging; an infrared receiving camera 3 used to receive the reflected light of the infrared fill light laser 4 on the target object and to perform imaging and output; a laser gimbal 5 used to adjust the horizontal and pitch angle rotation of the infrared fill light laser 4; a two-axis gimbal 6 used to adjust the pointing position of the entire multi-channel camera fusion device and to send back position information in real time, and a main control module 7 used to control the rotation of each module, collect and send back data; the color camera 1, the thermal imaging camera 2, the infrared fill light laser 4, the infrared receiving camera 3, the laser gimbal 5 and the main control module 7 are installed and fixed on the two-axis gimbal 6.
[0154] The color camera 1 and thermal imaging camera 2 are 40x continuous zoom cameras that can transmit focal length data. The irradiation position and laser spot size of the infrared fill light laser 4 can be adjusted automatically according to the focal length data of the infrared receiving camera 3 to ensure that the image at the irradiation position is located in the middle of the infrared receiving camera 3.
[0155] The above only describes the device composition and image fusion method of this patent under specific restrictions, but it should be understood that in addition to the fusion of color images into thermal images described in this patent, other multi-type and multi-directional fusion methods such as thermal images into color images, infrared images and color images, as well as the field of view conversion involved in the intermediate process, and the subsequent extensible label transmission are all within the scope of protection of this patent.
[0156] The present invention decomposes the fusion of multiple cameras into image fusion between two cameras, combines the image mapping relationship at the same focal length between cameras, and the mapping of the relationship between the surround view image and the pan / tilt position, to realize image feature calculation and image matching for specific spatial positions between multiple cameras with different imaging principles, overcomes the distinction and matching positioning between the specific spatial position and the camera imaging area in the small field of view telephoto mode, and realizes precise positioning, fast matching and image fusion of images of the interest area at the same spatial position between cameras, thereby enhancing the accuracy of monitoring.
[0157] The present invention synchronizes the label positions between different multi-channel cameras, realizes synchronized label binding and position display of multiple cameras even at different focal lengths, and thus realizes consistent monitoring view, label position transmission and positioning; by calculating the image mapping matrix between different types of cameras, the precise correspondence and fusion between the images collected by each camera are realized, the quality of image fusion is improved, and thus the accuracy and convenience of monitoring are improved.
[0158] The present invention realizes accurate matching and positioning of image positions between cameras by converting the image coordinates of camera two into the image coordinates of camera one and the absolute angular coordinates of camera two in sequence, and finally converting them back into the image coordinates of camera one. The algorithm is simple and the image matching and positioning effect is good. The image distortion problem caused by the change of device angle and the arc movement of the area of interest between camera images is solved, and the accuracy of image fusion is improved.
[0159] Although the specification has been described in detail, it should be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the invention as defined by the appended claims. In addition, the specific embodiments described are not intended to limit the scope of the invention, and those of ordinary skill in the art can easily understand based on the present invention that currently existing or later to be developed processes, machines, manufactures, material compositions, means, methods, or steps can perform substantially the same functions as the embodiments of the present invention or obtain substantially the same results. Therefore, the appended claims are intended to include such processes, machines, manufactures, material compositions, means, methods or steps within their scope.
Claims
1. A method for fusion of multi-channel cameras with variable focus, applied to multi-channel cameras and a two-axis pan / tilt head, wherein the multi-channel cameras include at least two different types of cameras, characterized in that: The image fusion between the multiple cameras is the mutual fusion of the images between the two cameras, wherein the image fusion of two different types of cameras includes the following steps: Step 1: Get the center position of the optical axis of camera 1 and camera 2 (X 1center , Y 1center ), and obtain the detector physical sizes (h1, w1), (h2, w2) of camera 1 and camera 2; Step 2: Calculate the image mapping matrices M1 and M2 of camera 1 and camera 2 at the same focal length f0; Step 3: Adjust the focal length of the camera lens to the intermediate focal length f mid , based on the rotation of the two-axis gimbal, establish a surround view image Src, and establish a corresponding relationship table M between the surround view image slices and the two-axis gimbal position; Step 4: Select the region of interest in the image interface of camera 2. The image coordinates of the four vertices of the region of interest are A(x r1 ,y r1 )、B(x r2 ,y r2 )、C(x r3 ,y r3 )、D(x r4 ,y r4 ), the four vertices of the interest region enclose an image Src0; Step 5: According to the image mapping matrix M2 in step 2, obtain the image coordinates A'(x1, y1), B'(x2, y2), C'(x3, y3), and D'(x4, y4) corresponding to the four vertices in the image Src0 in camera 1; Step 6: Convert the image coordinates corresponding to camera 1 in step 5 into the absolute angular coordinates corresponding to camera 1. Step 7: Convert the absolute angle coordinates in the camera 1 in step 6 into the image coordinates under the camera 1 image. The focal length used in the conversion process is f mid ; A”(Θ1,Φ1)→a'(X 1C ,Y 1C ) B”(Θ2,Φ2)→b’(X 2C ,Y 2C ) C”(Θ3,Φ3)→c’(X 3C ,Y 3C ) D”(Θ4,Φ4)→d’(X 4C ,Y 4C ); Step 8: According to the current two-axis gimbal angles Θy and Φy, obtain the image slice of camera 1 in the surround image Src, and extract the image coordinates a'(X 1C , Y 1C ), b'(X 2C , Y 2C ), c'(X 3C , Y 3C ), d'(X 4C , Y 4C ) and the image area Src1 enclosed by the camera 1 and the camera 2 to realize the transmission and positioning of the corresponding tag positions; Step 9: The image fusion between camera 1 and camera 2 is finally displayed as Src out =a×Src0+(1-a)×Src1 Where a is the fusion coefficient, and its value range is [0,1].
2. The method for fusion of multiple cameras with variable focus according to claim 1, characterized in that: The camera 1 and the camera 2 are respectively one of a thermal imaging camera, a white camera, a color camera, a black and white camera and an infrared camera.
3. The method for fusion of multiple cameras with variable focus according to claim 1, characterized in that: The image mapping matrices M1 and M2 in step 2 satisfy the following conditions Among them, (x, y) is the coordinate in the original image; (x', y') is the coordinate in the object image; A, B, C, D are the mapping matrix elements that control rotation, scaling and cropping; Tx, Ty are the translation amounts.
4. The method for fusion of multiple cameras with variable focus according to claim 1, characterized in that: The horizontal angle resolution of the correspondence table M between the surround image slices and the two-axis gimbal positions in step 3 satisfies fovx≤2arctan(w1 / (2f max )); vertical angle resolution satisfies fovy≤2arctan(h1 / (2f max )), where f max is the longest focal length of the camera; w1 is the width of the camera-detector; h1 is the height of the camera-detector.
5. A variable-focus multi-channel camera fusion device using the multi-channel camera fusion method according to any one of claims 1 to 4, characterized in that: Including color camera; thermal imaging camera; An infrared fill light laser used for infrared fill light to assist the color camera and thermal imaging camera in achieving clear imaging; an infrared receiving camera used to receive the reflected light of the infrared fill light laser on the target object and to perform imaging and output; a laser gimbal used to adjust the horizontal and pitch angles of the infrared fill light laser; a two-axis gimbal used to adjust the pointing position of the entire multi-channel camera fusion device and to send back position information in real time, and a main control module used to control the rotation of each module, collect and send back data; the color camera, thermal imaging camera, infrared fill light laser, infrared receiving camera, laser gimbal and main control module are installed and fixed on the two-axis gimbal.
6. A multi-channel camera fusion device with variable focus as described in claim 5, wherein the color camera and the thermal imaging camera are 40x continuous zoom cameras that can transmit focal length data back.
7. A multi-channel camera fusion device with variable focus as described in claim 6, wherein the irradiation position and laser spot size of the infrared fill light laser can be adjusted automatically according to the focal length data of the infrared receiving camera to ensure that the imaging of the irradiation position is located in the middle position of the infrared receiving camera.
