Infrared panoramic image stitching method, infrared panoramic monitoring device and electronic equipment

Abstract

The present invention discloses an infrared panoramic image stitching method, an infrared panoramic monitoring device and an electronic device, comprising: obtaining infrared images taken by various cameras, establishing a camera coordinate system corresponding to each camera, establishing a world coordinate system, using the origin of the world coordinate system as the center of the sphere and r as the radius to establish a spherical surface, where r is obtained by calibrating the camera, establishing an output plane in the world coordinate system, obtaining a mapping relationship between points on the output plane and points on the spherical surface, obtaining a first coordinate based on the mapping relationship, obtaining a RT relationship between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate based on the RT relationship, where the RT relationship is a coordinate conversion relationship between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, where the target pixel value is the pixel value of the point on the output plane, and obtaining an infrared panoramic image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an infrared panoramic image stitching method, an infrared panoramic monitoring device and an electronic device. Background Art

[0002] Existing infrared panoramic image generation technologies often obtain multiple infrared images and then stitch and fuse all the infrared images to obtain an infrared panoramic image.

[0003] However, existing infrared panoramic image generation technology has problems such as large data computing volume and slow image stitching speed. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides an infrared panoramic image stitching method, an infrared panoramic monitoring device, and an electronic device that can solve the problems of existing infrared panoramic image generation technologies, such as large data computational workload and slow image stitching speed.

[0005] According to the infrared panoramic image stitching method of the embodiment of the first aspect of the present invention, the method includes: obtaining infrared images taken by each camera; establishing a camera coordinate system corresponding to each of the cameras, and establishing a world coordinate system with the center of gravity of the origin of each of the camera coordinate systems as the origin; establishing a spherical surface with the origin of the world coordinate system as the center of the sphere and r as the radius, where r is obtained by calibrating the camera; establishing an output plane in the world coordinate system, obtaining a mapping relationship between a point on the output plane and a point on the spherical surface, and obtaining a first coordinate based on the mapping relationship, where the first coordinate is the spherical mapping point in the world coordinate system. The spherical mapping point is a point on the sphere corresponding to the point on the output plane; the RT relationship between the world coordinate system and the camera coordinate system is obtained, and according to the RT relationship, the first coordinate is converted into a second coordinate, the second coordinate is a coordinate point of the spherical mapping point in the camera coordinate system, and the RT relationship is a coordinate conversion relationship between the world coordinate system and the camera coordinate system; the second coordinate is converted into a pixel coordinate point, and the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the pixel value of the point on the output plane, to obtain an infrared panoramic image.

[0006] The infrared panoramic image stitching method according to the first embodiment of the present invention has at least the following beneficial effects:

[0007] By obtaining infrared images taken by each camera, establishing a camera coordinate system corresponding to each camera, using the center of gravity of the origin of each camera coordinate system as the origin, establishing a world coordinate system, using the origin of the world coordinate system as the center of the sphere, establishing a spherical surface, establishing an output plane in the world coordinate system, obtaining a mapping relationship between points on the output plane and points on the spherical surface, obtaining a first coordinate based on the mapping relationship, the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point is the point on the spherical surface corresponding to the point on the output plane, obtaining an RT relationship between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate based on the RT relationship, the second coordinate is the coordinate point of the spherical mapping point in the camera coordinate system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value is the pixel value of the point on the output plane, obtaining an infrared panoramic image, only calculating the pixel value corresponding to each point on the output plane, without processing and calculating all infrared image data, reducing the amount of data calculation, and increasing the image stitching speed.

[0008] According to some embodiments of the present invention, obtaining a mapping relationship between a point on the output plane and a point on the sphere, and obtaining a first coordinate based on the mapping relationship, wherein the first coordinate is a coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the sphere corresponding to the point of the output plane, includes: obtaining the coordinates of the four vertices P1, P2, P3 and P4 of the output plane in the world coordinate system; performing equal-interval sampling between P1, P2, P3 and P4 to obtain the coordinates of each point on the output plane in the world coordinate system; projecting each point on the output plane onto the sphere, and obtaining the first coordinate based on the projection relationship between each point on the output plane and the sphere.

[0009] According to some embodiments of the present invention, obtaining a mapping relationship between a point on the output plane and a point on the spherical surface, and obtaining a first coordinate according to the mapping relationship, wherein the first coordinate is a coordinate of a spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, comprises: taking N longitudinal lines and M transverse lines on the output plane, wherein the N longitudinal lines divide the output plane into equal parts laterally, and the M transverse lines divide the output plane into equal parts longitudinally, and the N longitudinal lines and the M transverse lines form N*M first intersection points; taking N longitudes and M latitudes on the spherical surface, wherein the N longitudinal lines divide the spherical surface into equal parts laterally, and the M transverse lines divide the spherical surface into equal parts longitudinally, and the N longitudes and the M transverse lines form N*M second intersection points; and matching the N*M first intersection points with the N*M second intersection points in sequence one by one to obtain the first coordinate.

[0010] According to some embodiments of the present invention, the mapping relationship between the points on the output plane and the points on the sphere is obtained, and a first coordinate is obtained according to the mapping relationship, where the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is the point on the sphere corresponding to the point of the output plane, including: establishing an inscribed circular plane in the sphere, the radius of the inscribed circular plane is the same as the radius of the sphere, and the circumscribed square plane of the inscribed circular plane is the output plane; obtaining the coordinates of each point on the inscribed circular plane in the world coordinate system, that is, the coordinates of the intersection of the circumscribed square plane and the inscribed circular plane in the world coordinate system, to obtain the first coordinate.

[0011] According to some embodiments of the present invention, the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the pixel value of the point on the output plane to obtain an infrared panoramic image, including: performing image fusion on the overlapping areas of the infrared images corresponding to all the pixel coordinate points to obtain a target infrared image, and each pixel coordinate point in the overlapping area corresponds to the pixel value of multiple infrared images; using the pixel value of the target infrared image corresponding to the pixel coordinate point as the target pixel value, and the target pixel value is the pixel value of the point on the output plane to obtain an infrared panoramic image.

[0012] According to some embodiments of the present invention, performing image fusion on the overlapping areas of the infrared images corresponding to all the pixel coordinate points to obtain a target infrared image includes: acquiring a low-light image corresponding to the infrared image; performing enhancement processing on the low-light image and the infrared image to obtain a transition low-light image and a transition infrared image; fusing the transition low-light image and the transition infrared image to obtain an enhanced infrared image; and performing image fusion on the overlapping areas of the enhanced infrared images corresponding to all the pixel coordinate points to obtain a target infrared image.

[0013] According to some embodiments of the present invention, the enhancement processing of the low-light image and the infrared image to obtain a transitional low-light image and a transitional infrared image includes the following steps: presetting a gamma lookup table; mapping the pixel values ​​of the low-light image to corresponding gamma-corrected pixel values ​​according to the gamma lookup table to obtain a transitional low-light image; and mapping the pixel values ​​of the infrared image to corresponding gamma-corrected pixel values ​​according to the gamma lookup table to obtain a transitional infrared image.

[0014] According to some embodiments of the present invention, the transitional low-light image and the transitional infrared image are fused to obtain an enhanced infrared image, including: decomposing the transitional low-light image into a base layer image and a detail layer image; decomposing the transitional infrared image into a base layer image and a detail layer image; fusing the base layer image of the transitional low-light image and the base layer image of the transitional infrared image to obtain a target base layer image; fusing the detail layer image of the transitional low-light image and the detail layer image of the transitional infrared image to obtain a target detail layer image; and reconstructing the target base layer image and the target detail layer image to obtain an infrared panoramic image.

[0015] An infrared panoramic monitoring device according to an embodiment of a second aspect of the present invention includes: a camera, the camera being configured to capture low-light images and infrared images at multiple angles;

[0016] An image processing module, wherein the output end of the camera is connected to the input end of the image processing module, and the image processing module executes the above-mentioned infrared panoramic image stitching method;

[0017] The communication module has an output end of the image processing module connected to an input end of the communication module, and the communication module is used to transmit the infrared panoramic image to a host end.

[0018] The infrared panoramic monitoring device according to the second embodiment of the present invention has at least the following beneficial effects:

[0019] By obtaining infrared images taken by each camera, establishing a camera coordinate system corresponding to each camera, using the center of gravity of the origin of each camera coordinate system as the origin, establishing a world coordinate system, using the origin of the world coordinate system as the center of the sphere, establishing a spherical surface, establishing an output plane in the world coordinate system, obtaining a mapping relationship between points on the output plane and points on the spherical surface, obtaining a first coordinate based on the mapping relationship, the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point is the point on the spherical surface corresponding to the point on the output plane, obtaining an RT relationship between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate based on the RT relationship, the second coordinate is the coordinate point of the spherical mapping point in the camera coordinate system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value is the pixel value of the point on the output plane, obtaining an infrared panoramic image, only calculating the pixel value corresponding to each point on the output plane, without processing and calculating all infrared image data, reducing the amount of data calculation, and increasing the image stitching speed.

[0020] According to an embodiment of the third aspect of the present invention, an electronic device includes: at least one processor; at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, the infrared panoramic image stitching method as described above is implemented.

[0021] An electronic device according to an embodiment of the third aspect of the present invention has at least the following beneficial effects:

[0022] By obtaining infrared images taken by each camera, establishing a camera coordinate system corresponding to each camera, using the center of gravity of the origin of each camera coordinate system as the origin, establishing a world coordinate system, using the origin of the world coordinate system as the center of the sphere, establishing a spherical surface, establishing an output plane in the world coordinate system, obtaining a mapping relationship between points on the output plane and points on the spherical surface, obtaining a first coordinate based on the mapping relationship, the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point is the point on the spherical surface corresponding to the point on the output plane, obtaining an RT relationship between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate based on the RT relationship, the second coordinate is the coordinate point of the spherical mapping point in the camera coordinate system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value is the pixel value of the point on the output plane, obtaining an infrared panoramic image, only calculating the pixel value corresponding to each point on the output plane, without processing and calculating all infrared image data, reducing the amount of data calculation, and increasing the image stitching speed.

[0023] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 This is a flow chart of the infrared panoramic image stitching method of the present invention;

[0026] Figure 2 Flowchart a of the infrared panoramic image stitching method of the present invention for obtaining a mapping relationship between a point on an output plane and a point on a spherical surface, and obtaining a first coordinate according to the mapping relationship;

[0027] Figure 3 Flowchart b of the infrared panoramic image stitching method of the present invention is to obtain a mapping relationship between a point on an output plane and a point on a spherical surface, and obtain a first coordinate according to the mapping relationship;

[0028] Figure 4Flowchart c of the infrared panoramic image stitching method of the present invention for obtaining a mapping relationship between a point on an output plane and a point on a spherical surface, and obtaining a first coordinate according to the mapping relationship;

[0029] Figure 5 A flowchart of the infrared panoramic image stitching method of the present invention, wherein the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the pixel value of the point on the output plane, to obtain the infrared panoramic image;

[0030] Figure 6 This is a flow chart of the infrared panoramic image stitching method of the present invention, in which overlapping areas of infrared images corresponding to all pixel coordinate points are fused to obtain a target infrared image;

[0031] Figure 7 A positional relationship diagram of a first infrared image and a second infrared image on an output plane of the infrared panoramic image stitching method of the present invention;

[0032] Figure 8 This is a functional block diagram of the infrared panoramic monitoring device of the present invention.

[0033] Reference numerals:

[0034] Camera 100,

[0035] Image processing module 200,

[0036] Communication module 300. DETAILED DESCRIPTION

[0037] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0038] In the description of the present invention, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0039] In the description of the present invention, "a plurality" refers to more than two. The use of "first" or "second" is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of the indicated technical features, or implicitly indicating the order of the indicated technical features.

[0040] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0041] Refer to the following Figures 1 to 8 An infrared panoramic image stitching method, an infrared panoramic monitoring device, and an electronic device according to embodiments of the present invention are described.

[0042] like Figure 1 As shown, the infrared panoramic image stitching method according to the first embodiment of the present invention includes the following steps:

[0043] S100, obtaining infrared images captured by each camera 100;

[0044] S200, establishing a camera coordinate system corresponding to each camera 100, and establishing a world coordinate system with the center of gravity of the origin of each camera coordinate system as the origin;

[0045] S300, establishing a spherical surface with the origin of the world coordinate system as the center of the sphere and r as the radius, where r is obtained by calibrating the camera 100;

[0046] S400: Establish an output plane in a world coordinate system, obtain a mapping relationship between a point on the output plane and a point on the spherical surface, and obtain a first coordinate based on the mapping relationship, where the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is the point on the spherical surface corresponding to the point on the output plane;

[0047] S500, obtaining a RT relationship between the world coordinate system and the camera coordinate system, and converting a first coordinate into a second coordinate according to the RT relationship, where the second coordinate is the coordinate point of the spherical mapping point in the camera coordinate system, and the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system;

[0048] S600 , converting the second coordinates into pixel coordinate points, taking the pixel values ​​of the infrared image corresponding to the pixel coordinate points as target pixel values, the target pixel values ​​being the pixel values ​​of the points on the output plane, and obtaining an infrared panoramic image.

[0049] By obtaining infrared images captured by each camera 100, establishing a camera coordinate system corresponding to each camera 100, establishing a world coordinate system with the center of gravity of the origin of each camera coordinate system as the origin, establishing a spherical surface with the origin of the world coordinate system as the center of a sphere, establishing an output plane in the world coordinate system, obtaining a mapping relationship between a point on the output plane and a point on the spherical surface, obtaining a first coordinate based on the mapping relationship, the first coordinate being the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point being the point on the spherical surface corresponding to the point on the output plane, obtaining a RT relationship between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate based on the RT relationship, the second coordinate being the coordinate of the spherical mapping point in the camera coordinate system, the RT relationship being the coordinate conversion relationship between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value being the pixel value of the point on the output plane, and obtaining an infrared panoramic image. Only the pixel value corresponding to each point on the output plane is calculated, without processing and calculating all infrared image data, thereby reducing the amount of data calculation and increasing the image stitching speed.

[0050] In some embodiments of the present invention, the method for determining the origin of the world coordinate system (x0, y0, z0) is as follows: the camera coordinate system of one camera 100 is selected as the transition world coordinate system, and then, based on the RT relationship between the central camera 100 and the other cameras 100, the origins of the camera coordinate systems of the remaining cameras 100 are converted to the transition world coordinate system using the following calculation formula:

[0051] [X,Y,Z]·R·T=[X′,Y′,Z′]

[0052] Where [X, Y, Z] is the origin coordinate of each camera coordinate system, [X′, Y′, Z′] is the coordinate of the origin coordinate of each camera coordinate system in the transition world coordinate system, R is the rotation matrix, and T is the translation vector;

[0053] The center of gravity of the origin of all camera coordinate systems in the transition world coordinate system is calculated using the following formula:

[0054]

[0055]

[0056]

[0057] Where N is the number of cameras 100, X OW is the X-axis coordinate of the center of gravity, Y OW is the Y-axis coordinate of the center of gravity, Z OW The Z-axis coordinate of the center of gravity, the coordinate of the center of gravity (X OW , Y Ow, Z Ow ) as the origin to establish the world coordinate system.

[0058] like Figure 2 As shown, in some embodiments of the present invention, in step S400, a mapping relationship between a point on the output plane and a point on the spherical surface is obtained, the mapping relationship being a straight line projection mapping relationship, and a first coordinate is obtained according to the straight line projection mapping relationship. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, including the following steps:

[0059] S410, obtaining the coordinates of the four vertices P1, P2, P3 and P4 of the output plane in the world coordinate system;

[0060] S420, performing equal-interval sampling between P1, P2, P3, and P4 to obtain the coordinates of each point on the output plane in the world coordinate system;

[0061] S430 , projecting each point on the output plane onto a spherical surface, and obtaining first coordinates based on a projection relationship between each point on the output plane and the spherical surface.

[0062] In some embodiments of the present invention, assuming that the vertical viewing angle width of the output plane is 2β and the horizontal viewing angle width is 2α, the coordinates of points P1, P2, P3, and P4 in the world coordinate system in the initial state are as follows:

[0063]

[0064] where p w1 is the coordinate of P1 in the world coordinate system, p w2 is the coordinate of P2 in the world coordinate system, p w3 is the coordinate of P3 in the world coordinate system, P w4 is the coordinate of P4 in the world coordinate system, r is the radius of the sphere, and the optimal value of r can be obtained during the stereo calibration of the camera 100 (the distance from the camera 100 to the chessboard), or it can be manually set as a parameter (i.e., setting the stitching distance).

[0065] The user needs to set the display resolution of the output plane (horizontal resolution / vertical resolution). After setting the display resolution of the output plane, the P in the world coordinate system can be w1 、P w2 、P w3 、P w4 The output plane is composed of the image output, in order to obtain the coordinates of the coordinate points on the entire output plane in the world coordinate system, P w1 、P w2 、p w3 、p w4The four points are sampled at equal intervals according to Δx, Δy, and Δz, and the coordinates of all points on the output plane in the world coordinate system are quantized.

[0066] When the user adjusts the posture angle, such as rotating around the X-axis, Y-axis, and Z-axis by angles ψ, Φ, and θ, the total rotation matrix R is equal to the product of the rotation matrices:

[0067] R=R(ψ)·R(Φ)·R(θ)

[0068] Among them, R(ψ) is the rotation matrix around the X axis, R(Φ) is the rotation matrix around the Y axis, and R(θ) is the rotation matrix around the Z axis.

[0069] According to the total rotation matrix, p can be obtained w1 、p w2 、P w3 、P w4 The coordinates after the attitude angle transformation.

[0070] Project all points in the output plane onto the sphere, and use the following calculation formula to obtain the coordinates of the corresponding points on the sphere in the world coordinate system, and get the first coordinate (X W , Y W , Z W ):

[0071]

[0072] Among them, X p The X-axis coordinate of the point on the output plane in the world coordinate system, p The Y-axis coordinate of the point on the output plane in the world coordinate system, Z p The Z-axis coordinate of the point on the output plane in the world coordinate system.

[0073] The second coordinate is obtained according to the RT relationship between the world coordinate system and each camera coordinate system.

[0074] like Figure 3 As shown, in some embodiments of the present invention, in step S400, according to some embodiments of the present invention, a mapping relationship between a point on the output plane and a point on the spherical surface is obtained, the mapping relationship is an equidistant cylindrical projection mapping relationship, and a first coordinate is obtained according to the equidistant cylindrical projection mapping relationship. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, including:

[0075] S410. Take N longitudinal lines and M transverse lines on the output plane. The N longitudinal lines divide the output plane into equal parts transversely, and the M transverse lines divide the output plane into equal parts longitudinally. The N longitudinal lines and the M transverse lines form N*M first intersections.

[0076] S420. Take N longitudes and M latitudes on the sphere, with the N longitudes dividing the sphere equally horizontally and the M horizontals dividing the sphere equally vertically. The N longitudes and M latitudes form N*M second intersections.

[0077] S430 , matching the N*M first intersection points with the N*M second intersection points in sequence one by one to obtain first coordinates.

[0078] like Figure 4 As shown, in some embodiments of the present invention, in step S400, a mapping relationship between a point on the output plane and a point on the spherical surface is obtained, the mapping relationship being a 3D straight line projection, and a first coordinate is obtained according to the 3D straight line projection mapping relationship. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, including the following steps:

[0079] S410, establishing an inscribed circular plane in the spherical surface, wherein the radius of the inscribed circular plane is the same as the radius of the spherical surface, and the circumscribed square plane of the inscribed circular plane is the output plane;

[0080] S420 , obtaining the coordinates of each point of the inscribed circular plane in the world coordinate system, that is, the coordinates of the intersection of the circumscribed square plane and the inscribed circular plane in the world coordinate system, to obtain a first coordinate.

[0081] In some embodiments of the present invention, the resolution of the output plane is set to w*h. According to the first quadrant of the Z-axis and Y-axis planes in the world coordinate system, the following calculation formula is used to calculate any point P on the output plane: w The coordinates of (m, n) in the world coordinate system are:

[0082]

[0083] Similarly, we can obtain the coordinates of the output plane in other quadrants and get the first coordinate according to the formula When R>r, it means that the point on the output plane is not in the inscribed circle plane and can be discarded directly.

[0084] When the inscribed circular plane is rotated, the coordinates of each point in the new inscribed circular plane in the world coordinate system can be obtained according to the rotation matrix.

[0085] like Figure 5 As shown, in some embodiments of the present invention, in step S600, the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the pixel value of the point on the output plane, to obtain the infrared panoramic image, including:

[0086] S610, performing image fusion on the overlapping areas of the infrared images corresponding to all pixel coordinate points to obtain a target infrared image, where each pixel coordinate point in the overlapping area corresponds to a pixel value of multiple infrared images;

[0087] S620 , taking the pixel value of the target infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value is the pixel value of the point on the output plane, and obtaining an infrared panoramic image.

[0088] In some embodiments of the present invention, Figure 7 As shown, the output plane corresponds to two infrared images: the first infrared image and the second infrared image. The first infrared image includes regions D1 and D2, and the second infrared image includes regions D3 and D4. Regions D1 and D4 correspond to different regions of the output plane, meaning they do not overlap. Regions D2 and D3 correspond to the same region on the output plane, meaning they overlap. The infrared images of regions D2 and D3 are fused to obtain a fused region infrared image. The infrared images of regions D1, D2, and D4 are sequentially overlaid on the output plane to obtain a target infrared image. The pixel value of the target infrared image corresponding to the pixel coordinate point is used as the target pixel value, which is the pixel value of the point on the output plane, to obtain an infrared panoramic image. Regions D2 and D3 are fused using a linear fusion algorithm, dynamically adjusted according to the fusion weight. The fusion weight is determined by the distance from the pixel point on the infrared image to the edge of the infrared image. The closer the distance, the smaller the fusion weight, while the farther the distance, the larger the fusion weight, ensuring a smooth transition between the overlapping regions. If the overlapping area of ​​the first infrared image and the second infrared image is too large, a mask image (binarized rectangular image) can be set to narrow the field of view of the infrared image obtained by the corresponding camera 100, reduce the number of fusion points, and improve fusion efficiency.

[0089] like Figure 6 As shown, in some embodiments of the present invention, in step S610, the overlapping areas of the infrared images corresponding to all pixel coordinate points are subjected to image fusion to obtain a target infrared image, including:

[0090] S611, acquiring a low-light-level image corresponding to the infrared image;

[0091] S612, performing enhancement processing on the low-light-level image and the infrared image to obtain a transitional low-light-level image and a transitional infrared image;

[0092] S613, fusing the transitional low-light image and the transitional infrared image to obtain an enhanced infrared image;

[0093] S614: Perform image fusion on the overlapping areas of the enhanced infrared images corresponding to all pixel coordinate points to obtain a target infrared image.

[0094] In some embodiments of the present invention, in step S614, the transitional low-light image and the transitional infrared image are fused using a Laplacian pyramid fusion method to obtain an enhanced infrared image, including:

[0095] Decomposing the transition low-light image into a base layer image and a detail layer image;

[0096] Decomposing the transition infrared image into a base layer image and a detail layer image;

[0097] The base layer image of the transitional low-light image and the base layer image of the transitional infrared image are fused to obtain a target base layer image;

[0098] The detail layer image of the transitional low-light image and the detail layer image of the transitional infrared image are fused to obtain a target detail layer image; the target base layer image and the target detail layer image are reconstructed to obtain an infrared panoramic image.

[0099] By performing a two-scale decomposition of the transitional low-light image and the transitional infrared image into a base layer image and a detail layer image, the base layer image is fused using a weighted coefficient method, and the detail layer image is fused using a weighted coefficient method. Finally, the fused images of each layer are reconstructed, quickly and efficiently fusing image information at each scale to obtain an infrared panoramic image with rich image information, high clarity, and high discernibility. GPU parallel acceleration is used for the mosaic fusion of infrared panoramic images, improving the efficiency of mosaic fusion.

[0100] In step S612, the low-light-level image and the infrared image are enhanced using a gamma image enhancement algorithm to obtain a transitional low-light-level image and a transitional infrared image, including the following steps:

[0101] Preset gamma lookup table;

[0102] According to the gamma lookup table, the pixel values ​​of the low-light image are mapped to the corresponding gamma-corrected pixel values ​​to obtain the transition low-light image;

[0103] According to the gamma lookup table, the pixel values ​​of the infrared image are mapped to the corresponding gamma-corrected pixel values ​​to obtain a transition infrared image.

[0104] The principle of the gamma image enhancement algorithm is as follows:

[0105] S100, normalization: converting pixel values ​​into real numbers between 0 and 1. The normalization calculation formula is:

[0106] f=(i+0.5) / 256, where f is the normalized pixel value and i is the pixel value

[0107] S200, pre-compensation: pre-compensate the normalized value. The calculation formula for pre-compensation is:

[0108] F = (1 / gamma) * f, where F is the pre-compensated pixel value

[0109] S300, denormalization: denormalize the pre-compensated real value into an integer value between 0 and 255. The calculation formula for denormalization is:

[0110] P=F*256-0.5, P is the denormalized pixel value

[0111] Any pixel value in low-light-level images and infrared images is limited to an integer between 0 and 255. The gamma value is set between 0 and 10 according to the enhancement intensity requirement. After normalization, pre-compensation, and denormalization operations, the corresponding calculation result is unique and also falls within the range of 0 to 255, which greatly reduces the amount of calculation, speeds up processing, and improves the recognizability of dark area images.

[0112] like Figure 8 As shown, the infrared panoramic monitoring device according to the embodiment of the second aspect of the present invention includes: a camera 100, an image processing module 200 and a communication module 300. The camera 100 is used to collect low-light images and infrared images at multiple angles. The output end of the camera 100 is connected to the input end of the image processing module 200. The image processing module 200 executes the infrared panoramic image stitching method as described above. The output end of the image processing module 200 is connected to the input end of the communication module 300. The communication module 300 is used to transmit the infrared panoramic image to the host end.

[0113] By obtaining the infrared images taken by each camera 100, a camera coordinate system corresponding to each camera 100 is established, and the center of gravity of the origin of each camera coordinate system is used as the origin to establish a world coordinate system. The origin of the world coordinate system is used as the center of the sphere to establish a spherical surface. An output plane is established in the world coordinate system, and a mapping relationship between a point on the output plane and a point on the spherical surface is obtained. According to the mapping relationship, a first coordinate is obtained, which is the coordinate of the spherical mapping point in the world coordinate system. The spherical mapping point is a point on the spherical surface corresponding to the point on the output plane. The RT relationship between the world coordinate system and the camera coordinate system is obtained. According to R T relationship, converts the first coordinate into the second coordinate, the second coordinate is the coordinate point of the spherical mapping point in the camera coordinate system, RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, converts the second coordinate into a pixel coordinate point, and uses the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value is the pixel value of the point on the output plane, and obtains an infrared panoramic image. Only the pixel values ​​corresponding to each point on the output plane are calculated, and there is no need to process and calculate all the infrared image data, which reduces the amount of data calculation, has a fast image stitching speed, and has good real-time monitoring and display performance on the host side.

[0114] Six cameras 100 are located on the same spherical surface. Their coverage area covers the upper hemisphere, enabling the capture of 360-degree infrared and low-light images. Fisheye cameras 100 offer a wider viewing angle. Other numbers of cameras 100 can be configured depending on the application scenario.

[0115] An electronic device according to an embodiment of the third aspect of the present invention includes: at least one processor; at least one memory for storing at least one program; and the above-mentioned infrared panoramic image stitching method is implemented when the at least one program is executed by the at least one processor.

[0116] By obtaining infrared images captured by each camera 100, establishing a camera coordinate system corresponding to each camera 100, establishing a world coordinate system with the center of gravity of the origin of each camera coordinate system as the origin, establishing a spherical surface with the origin of the world coordinate system as the center of a sphere, establishing an output plane in the world coordinate system, obtaining a mapping relationship between a point on the output plane and a point on the spherical surface, obtaining a first coordinate based on the mapping relationship, the first coordinate being the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point being the point on the spherical surface corresponding to the point on the output plane, obtaining a RT relationship between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate based on the RT relationship, the second coordinate being the coordinate of the spherical mapping point in the camera coordinate system, the RT relationship being the coordinate conversion relationship between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value being the pixel value of the point on the output plane, and obtaining an infrared panoramic image. Only the pixel value corresponding to each point on the output plane is calculated, without processing and calculating all infrared image data, thereby reducing the amount of data calculation and increasing the image stitching speed.

[0117] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in the relevant technical field without departing from the scope of the present invention.

Claims

1. Infrared panoramic image stitching method, characterized in that: include: Obtaining infrared images captured by each camera (100); Establishing a camera coordinate system corresponding to each of the cameras (100), and establishing a world coordinate system with the center of gravity of the origin of each of the camera coordinate systems as the origin; A spherical surface is established with the origin of the world coordinate system as the center of the sphere and r as the radius, wherein r is obtained by calibrating the camera (100); Establishing an output plane in the world coordinate system, obtaining a mapping relationship between a point on the output plane and a point on the spherical surface, and obtaining a first coordinate based on the mapping relationship, where the first coordinate is a coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane; Obtaining a RT relationship between the world coordinate system and the camera coordinate system, and converting a first coordinate into a second coordinate according to the RT relationship, where the second coordinate is a coordinate point of the spherical mapping point in the camera coordinate system, and the RT relationship is a coordinate conversion relationship between the world coordinate system and the camera coordinate system; Convert the second coordinate into a pixel coordinate point, Acquire a low-light-level image corresponding to the infrared image; performing enhancement processing on the low-light-level image and the infrared image to obtain a transitional low-light-level image and a transitional infrared image; fusing the transitional low-light level image and the transitional infrared image to obtain an enhanced infrared image; Performing image fusion on the overlapping areas of the enhanced infrared images corresponding to all the pixel coordinate points to obtain a target infrared image, wherein each pixel coordinate point in the overlapping area corresponds to a pixel value of multiple infrared images; The pixel value of the target infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the pixel value of the point on the output plane to obtain an infrared panoramic image.

2. The infrared panoramic image stitching method according to claim 1, characterized in that: The acquiring a mapping relationship between a point on the output plane and a point on the spherical surface, and obtaining a first coordinate according to the mapping relationship, wherein the first coordinate is a coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, includes: Obtain the coordinates of the four vertices P1, P2, P3 and P4 of the output plane in the world coordinate system; Performing equal-interval sampling between P1, P2, P3, and P4 to obtain the coordinates of each point on the output plane in the world coordinate system; Project each point on the output plane onto the spherical surface, and obtain the first coordinates according to the projection relationship between each point on the output plane and the spherical surface.

3. The infrared panoramic image stitching method according to claim 1, characterized in that: The acquiring a mapping relationship between a point on the output plane and a point on the spherical surface, and obtaining a first coordinate according to the mapping relationship, wherein the first coordinate is a coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, includes: Taking N longitudinal lines and M transverse lines on the output plane, the N longitudinal lines divide the output plane into equal parts laterally, and the M transverse lines divide the output plane into equal parts longitudinally, and the N longitudinal lines and the M transverse lines form N*M first intersections; Take N longitudes and M latitudes on the spherical surface, wherein the N longitudes divide the spherical surface into equal parts horizontally, and the M latitudes divide the spherical surface into equal parts vertically, and the N longitudes and M latitudes form N*M second intersections; The first coordinates are obtained by sequentially matching N*M first intersection points with N*M second intersection points one by one.

4. The infrared panoramic image stitching method according to claim 1, characterized in that: The step of obtaining a mapping relationship between a point on the output plane and a point on the spherical surface, and obtaining a first coordinate according to the mapping relationship, wherein the first coordinate is a coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, includes: An inscribed circular plane is established in the spherical surface, wherein the radius of the inscribed circular plane is the same as the radius of the spherical surface, and a circumscribed square plane of the inscribed circular plane is the output plane; The coordinates of each point of the inscribed circular plane in the world coordinate system are obtained, that is, the coordinates of the intersection of the circumscribed square plane and the inscribed circular plane in the world coordinate system, to obtain the first coordinates.

5. The infrared panoramic image stitching method according to claim 1, characterized in that: The step of enhancing the low-light-level image and the infrared image to obtain a transitional low-light-level image and a transitional infrared image comprises the following steps: Preset gamma lookup table; According to the gamma lookup table, the pixel values ​​of the low-light image are mapped to corresponding gamma-corrected pixel values ​​to obtain a transitional low-light image; According to the gamma lookup table, the pixel values ​​of the infrared image are mapped to corresponding gamma-corrected pixel values ​​to obtain a transition infrared image.

6. The infrared panoramic image stitching method according to claim 1, characterized in that: Fusing the transitional low-light level image and the transitional infrared image to obtain an enhanced infrared image includes: Decomposing the transitional low-light image into a base layer image and a detail layer image; Decomposing the transition infrared image into a base layer image and a detail layer image; fusing the base layer image of the transitional low-light image and the base layer image of the transitional infrared image to obtain a target base layer image; fusing the detail layer image of the transitional low-light level image and the detail layer image of the transitional infrared image to obtain a target detail layer image; The target base layer image and the target detail layer image are reconstructed to obtain an infrared panoramic image.

7. Infrared panoramic monitoring device, characterized in that: include: A camera (100), the camera (100) is used to collect low-light images and infrared images at multiple angles; an image processing module (200), wherein the output end of the camera (100) is connected to the input end of the image processing module (200), and the image processing module (200) executes the infrared panoramic image stitching method according to any one of claims 1 to 6; A communication module (300), wherein the output end of the image processing module (200) is connected to the input end of the communication module (300), and the communication module (300) is used to transmit the infrared panoramic image to a host end.

8. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the infrared panoramic image stitching method according to any one of claims 1 to 6 is implemented.

Images