Image aligning device and image aligning method

Abstract

To provide an image aligning device capable of adjusting positional deviation between a first image and a second image which are aligned with each other.SOLUTION: Automatic adjustment means automatically adjusts and aligns positions of a first image 13 and a second image 14 on the basis of shape information of an imaging object 21 present in the first image 13 and the second image 14. Image display means superimposes and displays the first image 13 and the second image 14 which are aligned with each other. When one of the first image 13 and the second image 14 is set to be a fixed image and the other is set to be a movable image, icon display means allows the image display means to display an icon for manual operation for moving the movable image. Image fine adjustment means changes a stereo parameter on the basis of an operation amount of the icon for manual operation, allows the image display means to superimpose in real time and display the movable image being moved by reflecting a change result of the stereo parameter and the fixed image.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an image alignment device and an image alignment method for aligning the positions of a first image and a second image.

Background Art

[0002] Conventionally, some position estimation display devices that estimate and display the positions of humans and the like use a visible camera and a thermal camera to image a position estimation object. Then, based on the shape information of an imaging target object such as a marker board, a technique for more accurately detecting the position of the position estimation object by aligning the images obtained by each camera has been considered (see, for example, Patent Document 1).

[0003] By the way, when imaging with a thermal camera, since it is necessary to store heat in the marker board, the marker board needs to have a certain thickness (for example, 2 mm). Note that the visible camera images only the surface of the marker board, while the thermal camera images an image including the plate thickness in addition to the surface of the marker board. For this reason, there is a problem that a deviation occurs by the plate thickness when aligning the images. That is, since the superimposed images do not completely match, conventionally, the images are aligned with an approximate value.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, even when the images are aligned with approximate values, there is a slight deviation between the position of the visible image obtained by imaging with a visible camera and the position of the thermal image obtained by imaging with a thermal camera. As a result, the temperature change portion shown in the thermal image (specifically, the portion where the color changes) is displayed shifted with respect to the visible image, and there is a problem that the temperature change portion cannot be specified.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide an image alignment device and an image alignment method capable of adjusting the deviation between the aligned first image and second image.

Means for Solving the Problems

[0007] In order to solve the above problems, the invention according to claim 1 is based on the shape information of a common imaging object in the first image obtained by imaging with a first camera and the second image obtained by imaging with a second camera. An automatic adjustment means for automatically adjusting and aligning the positions of the first image and the second image; an image display means for overlapping and displaying the first image and the second image aligned by the automatic adjustment means; When one of the first image and the second image is a fixed image and the other is a movable image, an icon display means for generating an icon for manual operation for moving the movable image and displaying it on the image display means to prompt manual operation by an operator; Image fine adjustment means for changing the stereo parameters based on the operation amount of the manual operation icon and overlapping and displaying the moved movable image and the fixed image in real time on the image display means while reflecting the change result of the stereo parameters. The gist is an image alignment device characterized by having.

[0008] In the invention according to claim 1, even if there is a deviation between the position of the aligned first image and the position of the second image, an operator can manually operate an icon for manual operation to move the movable image (the second image or the first image), thereby finely adjusting the deviation between the first image and the second image. As a result, the changed portion of the image shown in the movable image (or the fixed image) can be displayed without deviation with respect to the fixed image (or the movable image), and the superimposed image can be accurately displayed on the image display means.

[0009] The invention according to claim 2 is based on claim 1, wherein the imaging object is a marker installed on a plane, the first camera is a visible camera that acquires a visible image as the first image, the second camera is a thermal camera that acquires a thermal image as the second image, the visible camera and the thermal camera image the marker in a heated or cooled state, and the gist of the automatic adjustment means is to automatically adjust and align the positions of the visible image and the thermal image based on the shape information of the marker that is a common imaging object.

[0010] In the invention according to claim 2, an operator can manually operate an icon for manual operation to move the movable image (thermal image or visible image), thereby finely adjusting the deviation between the visible image and the thermal image. Further, since the marker is heated or cooled during alignment, the marker can be surely reflected in the thermal image acquired by imaging the marker with the thermal camera. As a result, alignment can be surely performed based on the shape information of the marker existing in the visible image and the thermal image.

[0011] The invention according to claim 3 is based on claim 2, wherein the gist of the marker is a marker board having a plurality of through holes.

[0012] In the invention according to claim 3, since both the visible image and the thermal image can be aligned using a single marker board, the position of the object to be positioned can be accurately detected. In addition, since the alignment control based on the shape information of the marker board becomes easy, the burden on the automatic adjustment means can be reduced.

[0013] The invention according to claim 4 is, in claim 1, characterized in that the icon display means generates, as the icons for manual operation, a rotation icon for moving the movable image in the rotational direction and a translation icon for moving the movable image in the translational direction, and causes the image display means to display them.

[0014] In the invention according to claim 4, by an operator operating the rotation icon or the translation icon to move the movable image (the second image or the first image), the deviation between the first image and the second image can be easily fine-tuned.

[0015] The invention according to claim 5 is, in any one of claims 1 to 4, characterized in that the icon display means further generates a reset icon for resetting the amount of change in the stereo parameters and returning the movable image to the initial state, and causes the image display means to display it.

[0016] In the invention according to claim 5, even if the deviation between the first image and the second image cannot be made sufficiently small despite moving the movable image, the movable image can be quickly returned to the initial state. Therefore, the operation of moving the movable image by operating the icons for manual operation can be easily repeated.

[0017] The invention according to claim 6 is, in any one of claims 1 to 4, characterized in that the icon display means further generates an imaging icon for imaging the object to be imaged and an automatic adjustment icon for performing automatic adjustment by the automatic adjustment means after imaging the object to be imaged, and causes the image display means to display them.

[0018] In the invention according to claim 6, by an operator operating the imaging icon, imaging of an imaging object can be performed according to the operator's will. Further, by the operator operating the automatic adjustment icon, alignment of the first image and the second image by the automatic adjustment means can be performed according to the operator's will.

[0019] The invention according to claim 7 is characterized in that, in any one of claims 1 to 4, the image display means numerically displays the amount of change in the stereo parameters.

[0020] In the invention according to claim 7, by an operator viewing the amount of change in the stereo parameters, the operator can intuitively know how much the movable image has moved.

[0021] The invention according to claim 8 includes: an imaging step of imaging a first image with a first camera and imaging a second image with a second camera; an automatic adjustment step of automatically adjusting and aligning the positions of the first image and the second image based on shape information of a common imaging object in the first image and the second image; an image display step of superimposing the first image and the second image aligned by the automatic adjustment step and displaying them on an image display means; an icon display step of generating an icon for manual operation for moving the movable image and displaying it on the image display means to prompt manual operation by an operator when one of the first image and the second image is a fixed image and the other is a movable image; a parameter change step of changing stereo parameters based on the operation amount of the icon for manual operation by manual operation; and an image fine adjustment step of reducing the deviation between the first image and the second image by superimposing the movable image moved reflecting the change result of the stereo parameters and the fixed image in real time and displaying them on the image display means.

[0022] In the invention according to claim 8, even if a deviation occurs between the position of the aligned first image and the position of the second image as a result of performing the automatic adjustment step, in the icon display step, an icon for manual operation is displayed on the icon display means, and the operator is prompted to perform a manual operation. Therefore, if the operator performs a manual operation to move the movable image (the second image or the first image), the stereo parameters are changed based on the operation amount of the icon. Then, by displaying the movable image moved in reflection of the change result of the stereo parameters and the fixed image (the first image or the second image) in an overlapped state, the deviation between the first image and the second image is reduced. As a result, the changed portion of the image shown in the movable image (or the fixed image) can be displayed without deviation with respect to the fixed image (or the movable image), and the overlapped image can be accurately displayed on the image display means.

Effect of the Invention

[0023] As described in detail above, according to the inventions according to claims 1 to 8, the deviation between the aligned first image and the second image can be adjusted.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0025] Hereinafter, an embodiment embodying the present invention will be described in detail with reference to the drawings.

[0026] As shown in FIG. 1, the position estimation display device 1 of the present embodiment is a device that estimates and displays the position of a human A1 (position estimation object). The position estimation display device 1 includes a visible camera 11 (first camera) and a thermal camera 12 (second camera). The visible camera 11 of the present embodiment is a fixed camera that images a human A1 in a room and acquires a visible image 13 (see FIGS. 2 and 4), which is a first image. Further, the thermal camera 12 of the present embodiment is a fixed camera that images a human A1 in a room and acquires a thermal image 14 (see FIGS. 2 and 5), which is a second image. Then, the visible camera 11 outputs the image data of the acquired visible image 13, and the thermal camera 12 outputs the image data of the acquired thermal image 14. Note that the visible image 13 is a color image, and the thermal image 14 is an infrared image. Further, the thermal image 14 is configured by filling the inside of a rectangular frame 14a (see FIG. 2) with different colors according to the temperature. The color of the thermal image 14 changes in the order of, for example, blue → yellow-green → yellow → red as the temperature increases.

[0027] Also, on the indoor floor surface 20 (plane), a marker board 21 (see FIGS. 4 and 5) as an imaging object is installed. The marker board 21 is a substantially rectangular plate-shaped steel plate having a certain thickness (for example, 0.5 mm or more and 20 mm or less (2 mm in this embodiment)) for heat storage. A plurality of circular through holes 22 are regularly arranged on the marker board 21. Specifically, each through hole 22 is arranged at the intersection of a plurality of first virtual lines inclined at 45° with respect to the long side and the short side of the marker board 21 and a plurality of second virtual lines inclined at 45° with respect to each first virtual line. The inner diameters of the respective through holes 22 are the same as each other. Also, the intervals between adjacent through holes 22 are the same as each other. And the inner diameter of the through hole 22 is the same as the interval between adjacent through holes 22. Note that the marker board 21 only needs to be placed on the floor surface 20 and does not need to be moved. However, when estimating the position of the human A1, the marker board 21 may be removed.

[0028] As shown in FIG. 1, the position estimation display device 1 includes an image alignment device 30. The image alignment device 30 includes a display 31 (image display means). The display 31 of the present embodiment is a display with a touch panel. Further, as shown in FIG. 2, a visible image 13 captured by the visible camera 11 and a thermal image 14 captured by the thermal camera 12 are displayed on the display screen 31a of the display 31. Furthermore, a human A1 (see FIG. 1) is shown in the images 13 and 14. Also, a target point 41 (see FIG. 16) is displayed at the feet of the human A1 shown in the images 13 and 14. The target point 41 is set at a predetermined point of the human A1, specifically, a point set on the boundary line between the human A1 and the floor surface 20 and between the left foot and the right foot of the human A1.

[0029] As shown in FIG. 2, in the upper right part of the display 31, as icons for manual operation, a rotation icon 50 for moving the movable image (the thermal image 14 in the present embodiment) in the rotation direction and a translation icon 60 for moving the movable image in the translation direction are displayed. The rotation icon 50 is composed of roll direction icons 55, 56, pitch direction icons 53, 54, and yaw direction icons 51, 52. The roll direction icons 55, 56 are icons for rotating the thermal image 14, that is, the posture of the thermal camera 12, in the roll direction (see "Roll" in FIG. 3). The pitch direction icons 53, 54 are icons for rotating the posture of the thermal camera 12 in the pitch direction (see "Pitch" in FIG. 3). The yaw direction icons 51, 52 are icons for rotating the posture of the thermal camera 12 in the yaw direction (see "Yaw" in FIG. 3). Note that each of the icons 51 to 56 is provided with a picture of a curved arrow. Specifically, the roll direction icon 55 is provided with a picture of an arrow turning from the left direction to the right direction, and the roll direction icon 56 is provided with a picture of an arrow turning from the right direction to the left direction. The pitch direction icon 53 is provided with a picture of an arrow turning from the upper direction to the back lower direction, and the pitch direction icon 54 is provided with a picture of an arrow turning from the upper direction to the front lower direction. The yaw direction icon 51 is provided with a picture of a substantially circular arrow rotating in the counterclockwise direction, and the yaw direction icon 52 is provided with a picture of a substantially circular arrow rotating in the clockwise direction.

[0030] Furthermore, in the upper right part of the display 31, at a position directly below the rotation icon 50, a reset icon 57 is displayed. The reset icon 57 is an icon for returning the movable image to the initial state by resetting the amount of movement of the movable image in the rotation direction. Note that the reset icon 57 is provided with the character "Reset".

[0031] As shown in FIG. 2, the translation icon 60 is arranged below the reset icon 57 in the upper right part of the display 31. The translation icon 60 consists of a left - direction icon 61, a right - direction icon 62, an up - direction icon 63, a down - direction icon 64, a back - direction icon 65, and a front - direction icon 66. The left - direction icon 61 is an icon for moving the thermal image 14, that is, the position of the thermal camera 12 in the left direction (the x - direction in FIG. 3). The right - direction icon 62 is an icon for moving the position of the thermal camera 12 in the right direction (the direction opposite to the x - direction). The up - direction icon 63 is an icon for moving the position of the thermal camera 12 in the up direction (the y - direction in FIG. 3). The down - direction icon 64 is an icon for moving the position of the thermal camera 12 in the down direction (the direction opposite to the y - direction). The back - direction icon 65 is an icon for moving the position of the thermal camera 12 in the back direction (the z - direction in FIG. 3). The front - direction icon 66 is an icon for moving the position of the thermal camera 12 in the front direction (the direction opposite to the z - direction). Note that each of the icons 61 - 66 is provided with a picture of a linear arrow. Specifically, the left - direction icon 61 is provided with a picture of an arrow extending in the left direction, the right - direction icon 62 is provided with a picture of an arrow extending in the right direction. The up - direction icon 63 is provided with a picture of an arrow extending in the up direction, the down - direction icon 64 is provided with a picture of an arrow extending in the down direction. The back - direction icon 65 is provided with a picture of an arrow extending in the back direction, and the front - direction icon 66 is provided with a picture of an arrow extending in the front direction.

[0032] Furthermore, in the upper right part of the display 31, a reset icon 67 is displayed at a position directly below the translation icon 60. The reset icon 67 is an icon for returning the movable image (thermal image 14) to the initial state by resetting the moving amount of the movable image in the translation direction. Note that the character "reset" is also given to the reset icon 67.

[0033] As shown in FIG. 2, an imaging icon 45, an automatic adjustment icon 46, and an end icon 47 are displayed in the lower right part of the display 31. The imaging icon 45 is an icon for imaging the marker board 21. The automatic adjustment icon 46 is an icon for automatically adjusting the positions of the visible image 13 and the thermal image 14. The end icon 47 is an icon for ending the manual operation of the movable image (the thermal image 14 in this embodiment).

[0034] Next, the electrical configuration of the image alignment device 30 will be described.

[0035] As shown in FIG. 1, the image alignment device 30 includes a personal computer (not shown), and the personal computer includes a control device 70 that comprehensively controls the entire device. The control device 70 is composed of a well-known computer including a CPU 71, a ROM 72, a RAM 73, etc. The display 31 and the keyboard 32 of the personal computer are electrically connected to the CPU 71. In this embodiment, the visible camera 11 and the thermal camera 12 are connected to the control device 70 via a USB (Universal Serial Bus) cable or a LAN (Local Area Network) cable, so that the visible camera 11 and the thermal camera 12 are electrically connected to the CPU 71. Note that the visible camera 11 and the thermal camera 12 may be wirelessly connected to the CPU 71. The visible image 13 acquired by the visible camera 11 and the thermal image 14 acquired by the thermal camera 12 are stored in the RAM 73. Also, a program for controlling the position estimation display device 1 is stored in the ROM 72.

[0036] Next, a method for aligning the visible image 13 and the thermal image 14 will be described.

[0037] First, a method for connecting the visible camera 11 and the thermal camera 12 will be described. As a prerequisite, it is necessary to know the positional relationship between the cameras 11 and 12 to be used and the floor surface 20. Therefore, a method for grasping the positional relationship between the cameras 11 and 12 and the floor surface 20 will be described first.

[0038] First, a marker board 21 (see FIGS. 4 and 5) having a plurality of through holes 22 is placed on the floor surface 20 in a heated state. Also, the CPU 71 of the control device 70 performs control to generate an imaging icon 45 (see FIG. 2) and display it on the display 31.

[0039] Next, in step S10 shown in FIG. 6, the CPU 71 determines whether the imaging icon 45 has been touched by the operator. If the imaging icon 45 has not been touched, the CPU 71 performs the process of step S10 again. Note that the process of step S10 is repeatedly executed until the imaging icon 45 is touched. When the imaging icon 45 is touched, the CPU 71 performs the process of the imaging step, captures a visible image 13 with the visible camera 11, and captures a thermal image 14 with the thermal camera 12. At this time, the visible camera 11 and the thermal camera 12 capture images with the marker board 21 in a heated state.

[0040] By the way, as shown in FIG. 7, the positional relationship between the marker board 21 and the visible camera 11 (or the thermal camera 12) is represented by stereo parameters consisting of a three-dimensional rotation matrix R and a translation vector t. Also, the marker board 21 has a coordinate system (marker coordinate system) centered on itself. On the other hand, the visible camera 11 (or the thermal camera 12) also has a coordinate system (camera coordinate system) centered on itself.

[0041] Therefore, the CPU 71 uses the stereo parameters (the three-dimensional rotation matrix R and the translation vector t) to convert the point (X, Y, Z) in the marker coordinate system based on the imaged marker board 21 into the point (x, y, z) on the camera coordinate system based on the visible camera 11 (or the thermal camera 12) (see Fig. 7). The point (x, y, z) on the camera coordinate system can be calculated using the determinant shown in the following equation (1). Here, the three-dimensional rotation matrix R is for rotating the point (X, Y, Z) on the marker coordinate system based on the imaged marker board 21 around the origin of the marker coordinates. The translation vector t is for translating the origin of the marker coordinates to the origin of the camera coordinate system based on the visible camera 11 (or the thermal camera 12). [Number]

[0042] Next, the CPU 71 calculates the height h from the floor surface 20 to the visible camera 11 (or the thermal camera 12), and the angle θ formed by the perpendicular line extending from the floor surface 20 to the visible camera 11 (or the thermal camera 12) and the line segment extending from the visible camera 11 (or the thermal camera 12) to the marker board 21. t Thereby, the positional relationship between the visible camera 11 (or the thermal camera 12) and the floor surface 20 is grasped (see Fig. 8). The positional relationship between the visible camera 11 and the floor surface 20 is represented as the floor surface vector F1 on the camera coordinate system, and the positional relationship between the thermal camera 12 and the floor surface 20 is represented as the floor surface vector F2 on the camera coordinate system. The length of the floor surface vector F1 is equal to the distance from the visible camera 11 to the floor surface 20, and the length of the floor surface vector F2 is equal to the distance from the thermal camera 12 to the floor surface 20. Also, the directions of the floor surface vectors F1 and F2 are opposite to the direction of the Z-axis of the marker coordinate system (that is, perpendicular to the floor surface 20).

[0043] Note that the floor vectors F1 and F2 are obtained as follows by using the geometric properties of vectors. First, in the camera coordinate system, a vector f in the direction opposite to the Z-axis perpendicular to the floor 20 is calculated. Note that e mz is the unit vector in the Z-axis direction in the marker coordinate system, and the vector f can be calculated by the following equation (2) based on the three-dimensional rotation matrix R.

Equation

[0044] Next, the angle θ formed by the perpendicular line extending from the floor 20 to the visible camera 11 (or the thermal camera 12) and the line segment extending from the visible camera 11 (or the thermal camera 12) to the marker board 21 is calculated using the vector f and the translation vector t. t Furthermore, the height h from the floor 20 to the visible camera 11 (or the thermal camera 12) is calculated using the translation vector t and the angle θ. First, the angle θ t can be calculated by the following equation (3) using the inner product of the vector f and the translation vector t. t

Equation

Equation

[0045] Also, the height h can be calculated by the following equation (4) using the translation vector t and the angle θ. t

Equation

Equation

[0046] Next, the floor vector F1, which is the vertical vector extending perpendicularly from the visible camera 11 to the floor 20, is calculated by scaling the vector f, and the floor vector F2, which is the vertical vector extending perpendicularly from the thermal camera 12 to the floor 20, is calculated. Note that the floor vectors F1 and F2 can be calculated by the following equation (5) based on the vector f and the height h.

Equation

[0047] As a result, the positional relationship between the visible camera 11 and the floor surface 20 is grasped by the floor surface vector F1, and the positional relationship between the thermal camera 12 and the floor surface 20 is grasped by the floor surface vector F2. Also, the positional relationship between the marker board 21 and the visible camera 11 and the positional relationship between the marker board 21 and the thermal camera 12 are estimated.

[0048] Furthermore, when the marker board 21 is imaged, the direct positional relationship between the cameras 11 and 12 is unknown, but the coordinate system (marker coordinate system) centered on the marker board 21 itself is shared. For this reason, it becomes possible to convert and compare the points seen from each of the cameras 11 and 12 to the points seen from the marker board 21 and further convert them to the points seen from the other camera.

[0049] Then, after imaging the marker board 21, the CPU 71 performs control to generate the auto-adjustment icon 46 (see FIG. 2) and display it on the display 31. Next, in step S20 shown in FIG. 6, the CPU 71 determines whether or not the auto-adjustment icon 46 has been touched by the operator. If the auto-adjustment icon 46 has not been touched, the CPU 71 performs the process of step S20 again. Note that the process of step S20 is repeatedly executed until the auto-adjustment icon 46 is touched. Then, when the auto-adjustment icon 46 is touched, the CPU 71 performs the process of step S30 (auto-adjustment step).

[0050] Specifically, the CPU 71 automatically adjusts and aligns the positions of the visible image 13 and the thermal image 14 based on the shape information of the common marker board 21 in the visible image 13 and the thermal image 14. That is, the CPU 71 has a function as an "automatic adjustment means". More specifically, the CPU 71 performs a calculation process of connecting the overhead coordinates of the cameras 11 and 12 via the marker board 21 to perform a process of aligning the visible image 13 and the thermal image 14. As shown in FIGS. 9 and 10, the "overhead coordinate system" refers to a coordinate system obtained by vertically overlooking the cameras 11 and 12, and is represented as an xy coordinate system with the same direction as the cameras 11 and 12 as the y-axis and the direction orthogonal to the y-axis as the x-axis. Further, the "process of aligning the visible image 13 and the thermal image 14" refers to a process of grasping the positional relationship between the cameras 11 and 12 and converting the point (x2, y2) in the overhead coordinate system of the thermal camera 12 to the point (x1, y1) in the overhead coordinate system of the visible camera 11. Note that a process of converting the point in the overhead coordinate system of the visible camera 11 to the point in the overhead coordinate system of the thermal camera 12 may be performed, or a process of converting both the point in the overhead coordinate system of the visible camera 11 and the point in the overhead coordinate system of the thermal camera 12 to the point in another overhead coordinate system may be performed.

[0051] In addition, since the overhead coordinate systems of the respective cameras 11 and 12 belong to the same floor surface 20, the conversion between the overhead coordinate systems can be performed by a three-dimensional rotation matrix R and a translation matrix T based on an affine transformation. Specifically, by using the properties of the marker coordinate system described above, an affine transformation matrix A for performing coordinate transformation is obtained by the following procedure.

[0052] First, the floor vectors F1 and F2 of the respective cameras 11 and 12 are inversely transformed by the following equation (6) to be transformed into the floor vectors F M1 , F M2 in the marker coordinate system.

Equation

[0053] Next, the unit vectors e z1 , e z2is converted into the direction vectors d M1 , d M2 of cameras 11 and 12 in the bird's-eye coordinate system as seen from the marker coordinate system using the following equation (7). M1 , d M2 .

Equation

Equation

[0054] Furthermore, the translation matrix T is calculated as follows. First, let the unit vector on the marker coordinate system be e My , and let the angle formed by the direction vector d M1 and the unit vector e My be θ T . Then, the angle θ T is calculated by the following equation (9). My and the direction vector d M1 and the unit vector e My be θ T . Then, the angle θ T is calculated by the following equation (9).

Equation

Equation

[0055] Furthermore, when converting the marker coordinate system to a coordinate system with the same origin as the marker coordinate system and having coordinate axes parallel to the bird's-eye coordinate system for the visible camera 11, the affine transformation matrix A1 is calculated by the following equation (11).

Equation

[0056] Also, on the above coordinate system, the translation vector t = (t1, t2, t3) from the thermal camera 12 to the visible camera 11 is calculated by the following equation (12).

Equation

[0057] Then, the translation matrix T is calculated by the following equation (13).

Equation

[0058] Furthermore, the three-dimensional rotation matrix R is calculated as follows. First, the direction vector d M1 and the direction vector d M2 The angle θ formed by them R Let it be θ R Then, the angle θ

Equation

[0059] And the three-dimensional rotation matrix R is calculated by the following equation (15).

Equation

[0060] Next, an affine transformation matrix A obtained by combining the three-dimensional rotation matrix R and the translation matrix T is obtained by the following equation (16).

Equation

Equation

[0061] Then, the CPU 71 performs the process of step S40 (image display step), overlays (combines) the visible image 13 and the thermal image 14 aligned in step S30 (automatic adjustment step), and displays them on the display 31 (see FIG. 11). In this embodiment, the thermal image 14 is overlaid on the visible image 13 and displayed on the display 31. In FIG. 11, the thermal image 14 is shown by a line thinner than the line indicating the visible image 13.

[0062] Also, the CPU 71 performs the process of the icon display step. Specifically, the CPU 71 sets the visible image 13, which is one of the visible image 13 and the thermal image 14, as a fixed image, and sets the thermal image 14, which is the other image, as a movable image. Then, the CPU 71 generates a rotation icon 50 and a translation icon 60, which are icons for manual operation to move the movable image, and performs control to display them on the display 31 (see FIG. 2). Thereby, manual operation (touch operation) by the operator is promoted. That is, the CPU 71 has a function as "icon display means". Also, the CPU 71 generates reset icons 57, 67 to return to the initial state, which is the state before the thermal image 14 is moved, and performs control to display them on the display 31 (see FIG. 2).

[0063] After step S40 (image display step) and before steps S70 - S90 (parameter change step) or steps S100 - S120 (parameter change step), the operator performs the process of step S50 (misalignment determination step). Specifically, the operator determines whether there is any misalignment between the superimposed visible image 13 and thermal image 14. If the operator determines that there is no misalignment, in step S60, the CPU 71 determines whether the end icon 47 (see FIG. 2) has been touched by the operator. If the end icon 47 has not been touched, the CPU 71 performs the process of step S50 again. Note that the process of step S50 is repeatedly executed until the end icon 47 is touched. On the other hand, if the end icon 47 has been touched, the CPU 71 ends the processing here without performing the parameter change step and image fine - tuning step.

[0064] Also, in step S50, if the operator determines that there is a misalignment between the visible image 13 and the thermal image 14 (see FIG. 11), the CPU 71 performs the process of step S70 or step S100. For example, when the operator touches and operates the rotation icon 50, which is an icon for manual operation, the CPU 71 performs the process of step S70. In step S70, the CPU 71 changes the stereo parameters based on the operation amount of the rotation icon 50 (the number of touch operations in this embodiment). Specifically, the CPU 71 changes the stereo parameters by the operation amount of the touched rotation icon 50 in the operation direction indicated by the touched rotation icon 50 for the thermal image 14. In other words, the CPU 71 selects the direction and angle of rotation of the thermal camera 12.

[0065] Note that the rotation operation is performed by specifying the rotation angle for any rotation axis (see Figure 3) among Roll, Pitch, and Yaw. Each rotation angle can be adjusted within the range of -180° to +180° (see Figure 12). Along with the specification of the rotation angle, the three-dimensional rotation matrix R is updated. The procedure for updating (changing) the three-dimensional rotation matrix R is as follows.

[0066] First, let the rotation angle be θ and the rotation axis be n = (n x , n y , n z ). Note that n is determined by the type of the touched icon and specifically has the following values. Roll direction icons 55, 56 → n = (0, 0, 1) Pitch direction icons 53, 54 → n = (1, 0, 0) Yaw direction icons 51, 52 → n = (0, 1, 0)

[0067] Next, the CPU 71 performs the process of step S80 to create the three-dimensional rotation matrix R m representing the input rotation. The three-dimensional rotation matrix R m corresponding to the input touch operation is calculated by the following equation (18).

Equation

[0068] Then, the CPU 71 performs the process of step S90 to obtain the new rotated matrix. Since the current three-dimensional rotation matrix is the three-dimensional rotation matrix R, the new three-dimensional rotation matrix R new after the touch operation is calculated by the following equation (19).

Equation

[0069] Furthermore, by replacing the existing three-dimensional rotation matrix R with the three-dimensional rotation matrix R new , the update (change) is completed.

[0070] Also, when an operator touches and operates the translation icon 60, which is an icon for manual operation, the CPU 71 performs the process of step S100. In step S100, the CPU 71 changes the stereo parameters based on the operation amount of the translation icon 60 (the number of touch operations in this embodiment). Specifically, the CPU 71 changes the stereo parameters of the thermal image 14 by the operation amount of the translation icon 60 that has been touched and operated in the operation direction indicated by the translation icon 60 that has been touched and operated. In other words, the CPU 71 selects the direction in which the thermal camera 12 is to be translated and the amount of movement for the translation.

[0071] Note that the translation operation is performed by specifying the amount of movement for moving the thermal camera 12 in an arbitrary direction (see FIG. 3) among the x-axis, y-axis, and z-axis as viewed from the thermal camera 12. Each amount of movement can be adjusted within the range of -1000 mm to +1000 mm (see FIG. 12). And with the specification of the amount of movement, the translation vector t is updated. The procedure for updating (changing) the translation vector t is as follows.

[0072] First, let the amount of movement be Δt and the movement direction be e. Note that e is determined by the type of icon that has been touched and operated. For example, when the left direction icon 61 is touched and operated to move the thermal camera 12 in the positive direction of the x-axis, the value of e is calculated by the following equation (20).

Equation

[0073] Also, when the upward direction icon 63 is touched and operated to move the thermal camera 12 in the positive direction of the y-axis, the value of e is calculated by the following equation (21).

Equation

[0074] Furthermore, when the depth direction icon 65 is touched to move the thermal camera 12 in the positive direction of the z-axis, the value of e is calculated by the following equation (22).

Equation

[0075] Also, when moving in the negative direction of each axis, the "1" in each matrix is replaced with "-1". Next, the CPU 71 performs the process of step S110 to create a translation vector t m representing the input movement amount. Note that the translation vector t m corresponding to the input touch operation is calculated by the following equation (23).

Equation

[0076] Then, the CPU 71 performs the process of step S120 to obtain the new moved vector. Since the current translation vector is the translation vector t, the new translation vector t new after the touch operation is calculated by the following equation (24).

[0077]

Equation

[0078] Furthermore, by replacing the existing t with t new , the update (change) is completed.

[0079] Note that after the end of step S90 or step S120, the CPU 71 performs the process of the image fine adjustment step. In the image fine adjustment step, the CPU 71 uses the change result of the stereo parameters (the three-dimensional rotation matrix R new or the translation vector t newThe thermal image 14 moved in accordance with (0) and the visible image 13 are superimposed in real time and displayed on the display 31, thereby performing control to reduce the deviation between the visible image 13 and the thermal image 14. That is, the CPU 71 has a function as an “image fine adjustment means”. In other words, the CPU 71 performs control to move the thermal image 14 as the stereo parameters are changed. At this time, the visible image 13 and the thermal image 14 are displayed on the display 31 while being superimposed. Further, the CPU 71 quantifies the amount of change (adjustment amount) of the stereo parameters and performs control to display it on the display 31 in real time (see FIG. 12). Thereafter, the CPU 71 performs the process of step S40 again.

[0080] Also, when the operator touches the reset icons 57, 67, the CPU 71 resets the amount of change of the stereo parameters and performs control to return to the initial state which is the state before moving the thermal image 14. At this time, the CPU 71 quantifies the adjustment amount of the stereo parameters (sets it to “0”) and performs control to display it on the display 31 (see FIG. 13). Also in this case, the CPU 71 performs the process of step S40 again.

[0081] Next, a method for estimating and displaying the position of the human A1 based on the cameras 11, 12 will be described. In the “process of aligning the visible image 13 and the thermal image 14” in step S30, the points in the overhead coordinate system of the thermal camera 12 are converted into the points in the overhead coordinate system of the visible camera 11. For this reason, in the present embodiment, the position of the human A1 reflected in the visible image 13 captured by the visible camera 11 is estimated and displayed. Specifically, based on the calculated floor vector F1, the position of the target point 41 on the overhead coordinate system based on the visible camera 11 is obtained (see FIG. 14). Note that the target point 41 is set at the feet of the human A1.

[0082] Note that in order to perform position estimation, the internal camera parameters of the visible camera 11 are required. The internal camera parameters are the focal length f of the visible camera 11 x , f y and the optical center c x , c yIt is a matrix consisting of the following. According to the following formula (25), a point (x, y, z) in the camera coordinate system can be converted into a point (a, b) on the image plane.

Number

[0083] Also, the camera internal parameters can be calculated by functions such as the CalibrateCamera function provided in OpenCV (Open Source Computer Vision Library) or by calculation from the data sheet of the visible camera 11. Then, the CPU 71 stores the information of the camera internal parameters of the visible camera 11 in the RAM 73.

[0084] Next, the CPU 71 sets a predetermined point set at the feet of the human A1 reflected in the captured visible image 13 as the target point 41. Specifically, the CPU 71 in the present embodiment has a function as a learning unit. The learning unit learns in advance how to recognize the setting location (the feet of the human A1) of the target point 41 based on the image data of the visible image 13 acquired by the visible camera 11, and obtains a learning result. Then, the learning unit stores the data indicating the obtained learning result in the RAM 73. Furthermore, the CPU 71 determines the setting method of the target point 41 based on the learning result stored in the RAM 73, and causes the CPU 71 to set the target point 41 according to the determined content.

[0085] Next, the CPU 71 estimates the position of the target point 41 in the bird's-eye view coordinate system based on the grasped position relationship information and the internal parameter information stored in the RAM 73 with the visible camera 11 as the reference. Specifically, as shown in FIGS. 15 and 16, the position (x, y) of the target point 41 in the coordinate system with the visible camera 11 as the reference is calculated as follows. First, by imaging the human A1 or the like, the target point (u, v) shown in FIG. 16 is determined. This target point (u, v) is assumed to exist on the bird's-eye view plane.

[0086] Then, a three-dimensional direction vector d for the target point (u, v) is calculated by the following equation (26).

Equation

Equation

[0087] Furthermore, after calculating the straight-line distance d to human A1 by the following equation (28), the distance l d on the floor 20 is calculated by the following equation (29).

Equation

Equation

[0088] Then, after calculating D by scaling the direction vector d by the following equation (30), the vector g is calculated by the following equation (31).

Equation

Equation

[0089] After that, using equations similar to equations (27) to (31), for the unit vector e cz in the z-axis direction on the camera coordinate system, the vector g y is obtained (specifically, the direction vector d in equations (27) to (31) is replaced with the unit vector e z , and the vector g is replaced with the vector g yis read as). As shown in Fig. 15, the vector g y is parallel to the y-axis of the overhead coordinate system in the camera coordinate system.

[0090] Next, using the following equation (32), the vector g y rotated 90° around the floor vector F1 is obtained as the vector g x .

Equation

Equation

[0091] Next, the angle θ formed by the vector g and the vector g y is calculated using the following equations (34) and (35).

Equation

Equation

[0092] As a result, from the distance l d and the angle θ, the position (x, y) of the target point 41 is obtained in polar coordinate representation. Therefore, (x, y) is x = l d sin θ, y = l d cos θ. Thus, the position of the human A1 with respect to the visible camera 11 is estimated.

[0093] After that, the CPU 71 causes the display 31 to display the visible image 13 of the overhead plane, and performs a process of displaying an icon indicating the target point 41 on the visible image 13. Thereby, the estimated position of the human A1 is displayed.

[0094] Therefore, according to the present embodiment, the following effects can be obtained.

[0095] (1) Conventionally, when a deviation occurred between the aligned visible image 13 and the thermal image 14 (see FIG. 11), the deviation could not be adjusted. On the other hand, in the image alignment device 30 of the present embodiment, even if a deviation occurs between the aligned visible image 13 and the thermal image 14, the operator can manually operate the rotation icon 50 or the translation icon 60 to move the thermal image 14, thereby finely adjusting the deviation. As a result, the temperature change portion shown in the thermal image 14 (for example, the portion of the through hole 22 where the color changes in the order of blue → yellow - green → yellow → red toward the edge) can be displayed without deviation with respect to the visible image 13, so that the superimposed image can be accurately displayed on the display 31. Therefore, by using the position estimation display device 1 in this state, the position of the target point 41 can be accurately estimated, and based on the estimated position of the target point 41, the position of the human A1 can be detected with higher accuracy.

[0096] (2) In the present embodiment, since the thermal camera 12 captures an image with the marker board 21 heated, the marker board 21 can be surely reflected in the captured thermal image 14. However, if the marker board 21 is too thin, the temperature of the entire marker board 21 will immediately increase during heating and the temperature difference will become small, making it difficult to distinguish. Therefore, since the marker board 21 of the present embodiment has a certain thickness (for example, 1 mm or more), the occurrence of the above - mentioned problem can be suppressed.

[0097] (3) In the present embodiment, the CPU 71 uses the change result of the stereo parameters (the three - dimensional rotation matrix R new or the translation vector t newThe thermal image 14 moved in accordance with (0) and the visible image 13 are superposed in real time and displayed on the display 31. For this reason, by the operator checking the display 31, not only the state of the images 13 and 14 after alignment but also the state of the images 13 and 14 during alignment can be checked.

[0098] (4) In the present embodiment, based on the vertical vectors (floor vectors F1, F2) as the positional relationship information and the information on the internal camera parameters of the cameras 11 and 12, the position of the target point 41 in the overhead coordinate system based on the cameras 11 and 12 is estimated. Thereby, just by installing the marker board 21 on the floor surface 20 and imaging it, the position of the human A1 can be estimated relatively easily and accurately. Moreover, after grasping the positional relationship, the position of the human A1 can be accurately estimated without installing the marker board 21.

[0099] (5) In the present embodiment, even if both the "height h" and the "angle θ t " of the cameras 11 and 12 are unknown, the cameras 11 and 12 can be used as cameras for position estimation, and moreover, accurate position estimation can be performed.

[0100] Note that the above embodiment may be modified as follows.

[0101] · In the above embodiment, the CPU 71 controls to reduce the deviation between the visible image 13 and the thermal image 14 by moving the thermal image 14, which is the movable image among the visible image 13 and the thermal image 14. However, the CPU 71 may control to reduce the deviation between the visible image 13 and the thermal image 14 by moving the visible image 13. That is, the visible image 13 may be used as the movable image.

[0102] · In the above embodiment, the visible camera 11 was used as the first camera, and the thermal camera 12 was used as the second camera. However, it may be changed to a different combination from the visible camera 11 and the thermal camera 12. For example, two visible cameras 11 may be used as the first camera and the second camera, respectively. Also, the visible camera 11 may be used as the first camera, and a depth camera capable of measuring the depth of an image may be used as the second camera. Further, the visible camera 11 may be used as the first camera, and a spectral camera may be used as the second camera.

[0103] · In the above embodiment, when aligning the visible image 13 and the thermal image 14, the visible camera 11 and the thermal camera 12 were imaging the marker board 21 in a heated state. However, the visible camera 11 and the thermal camera 12 may image the marker board 21 in a cooled state. Also, the thermal camera 12 may image the marker board 21 in a heated or cooled state, while the visible camera 11 may image the marker board 21 without heating or cooling it.

[0104] · In the above embodiment, the visible camera 11 and the thermal camera 12 were imaging the marker board 21 installed on the indoor floor surface 20. However, the visible camera 11 and the thermal camera 12 may image the marker board 21 held by an operator.

[0105] ·In the above-described embodiment, the substantially rectangular marker board 21 was used as a common imaging object in the visible image 13 and the thermal image 14. However, a marker board having a square shape, a parallelogram shape, a rhombus shape, a trapezoid shape, a circular shape, an elliptical shape, or the like may be used as the common imaging object. Further, instead of the marker board 21, a three-dimensional object such as a protrusion existing on the floor surface 20 may be used as the common imaging object (marker). Furthermore, although the marker board 21 was made of a steel plate, it may be made of a plate material made of another metal material such as aluminum or copper, or may be made of a plate material made of another material such as resin or ceramic. Also, although the marker board 21 had circular through-holes 22, it may have through-holes having a square shape, a rectangular shape, a parallelogram shape, a rhombus shape, a trapezoid shape, an elliptical shape, or the like. Also, each of the through-holes 22 was arranged on the intersections of a plurality of first virtual lines inclined at 45° with respect to the long side and the short side of the marker board 21 and a plurality of second virtual lines inclined at 45° with respect to each of the first virtual lines. However, if they are arranged regularly, there is no particular limitation. Note that the marker board 21 may not have the through-holes 22.

[0106] ·In the above-described embodiment, by operating the rotation icon 50 (rotation operation), the posture of the thermal camera 12 was moved in the roll direction, the pitch direction, and the yaw direction, and by operating the translation icon 60 (translation operation), the position of the thermal camera 12 was moved in the parallel direction. However, the thermal camera 12 may be moved only by operating the rotation icon 50, or the thermal camera 12 may be moved only by operating the translation icon 60. That is, the manual operation icon may include only one of them.

[0107] ·In the above-described embodiment, the number of touch operations of the manual operation icons (rotation icon 50, translation icon 60) was used as the operation amount of the icons 50, 60. However, the operation time of the icons 50, 60 when the icons 50, 60 are long-pressed may be used as the operation amount of the icons 50, 60.

[0108] · In step S60 of the above-described embodiment, the alignment between the visible image 13 and the thermal image 14 by manual operation was terminated upon the touch operation of the end icon 47 by the operator. However, when there has been no touch operation of any of the end icon 47, the rotation icon 50, the translation icon 60, and the reset icons 57 and 67 for a certain period, it may be regarded that the alignment has ended, and the alignment between the visible image 13 and the thermal image 14 by manual operation may be terminated.

[0109] · The image alignment device 30 of the above-described embodiment was configured to correct the deviation when a deviation occurred between the visible image 13 (first image) and the thermal image 14 (second image) due to the plate thickness of the marker board 21. However, the image alignment device may be configured to correct the deviation when a deviation occurs between the first image and the second image due to an image deviation specific to the first camera or the second camera, or due to distortion of the lens of the first camera or the second camera.

[0110] · In the above-described embodiment, the visible camera 11 and the thermal camera 12 transmitted image data to the control device 70 via a USB cable, but image data may be transmitted to the control device 70 using other means. For example, the visible camera 11 and the thermal camera 12 may transmit image data to the control device 70 via communication means such as Bluetooth (registered trademark of Bluetooth SIG, Inc.), infrared communication, or an Internet line (such as a telephone line).

[0111] · In the above-described embodiment, the position estimation display device 1 was configured to estimate and display the position of the human A1. However, the position estimation display device 1 may be configured to estimate and display the position of the flame at the fire occurrence location, the position of the hot air leakage from equipment such as a painting drying furnace, etc. Further, the position estimation display device 1 may be configured to estimate and display the position of a living thing such as an animal or the position of an inanimate object such as an automobile. Examples of living things include animals such as cows, horses, pigs, sheep, and goats.

[0112] Next, in addition to the technical idea described in the claims, the technical ideas grasped by the above-described embodiments are listed below.

[0113] (1) In claim 8, a deviation determination step of determining whether there is a deviation between the first image and the second image is performed after the image display step and before the parameter change step, and when it is determined in the deviation determination step that there is no deviation, the parameter change step and the image fine adjustment step are not performed. An image alignment method characterized by this.

[0114] (2) A position estimation display device including the image alignment device according to any one of claims 1 to 7.

Explanation of reference numerals

[0115] 11... Visible camera as the first camera 12... Thermal camera as the second camera 13... Visible image as the first image 14... Thermal image as the second image 20... Floor surface as a plane 21... Marker board as an imaging object 22... Through hole 30... Image alignment device 31... Display as an image display means 45... Imaging icon 46... Automatic adjustment icon 50... Rotation icon as an icon for manual operation 57, 67... Reset icon 60... Translation icon as an icon for manual operation 71... CPU as an automatic adjustment means, icon display means, and image fine adjustment means

Claims

1. Automatic adjustment means for automatically adjusting and aligning the positions of the first image and the second image based on the shape information of a common imaging object in the first image captured by the first camera and the second image captured by the second camera; Image display means for superimposing and displaying the first image and the second image aligned by the automatic adjustment means; Icon display means for generating an icon for manual operation to move the movable image and displaying it on the image display means to prompt manual operation by an operator when one of the first image and the second image is a fixed image and the other is a movable image; Image fine adjustment means for changing the stereo parameters based on the operation amount of the icon for manual operation and superimposing and displaying the movable image and the fixed image moved in real time while reflecting the change result of the stereo parameters on the image display means An image alignment device, characterized by comprising the above.

2. The imaging object is a marker installed on a plane, The first camera is a visible camera that acquires a visible image as the first image, The second camera is a thermal camera that acquires a thermal image as the second image, The visible camera and the thermal camera image the marker in a heated or cooled state, The automatic adjustment means automatically adjusts and aligns the positions of the visible image and the thermal image based on the shape information of the marker, which is a common imaging object. The image alignment device according to claim 1, characterized by the above.

3. The image alignment device according to claim 2, characterized in that the marker is a marker board having a plurality of through holes.

4. The icon display means generates, as the icon for manual operation, a rotation icon for moving the movable image in the rotation direction and a translation icon for moving the movable image in the translation direction and displays them on the image display means. The image alignment device according to claim 1, characterized by the above.

5. The icon display means further generates a reset icon for resetting the change amount of the stereo parameters and returning the movable image to the initial state and displays it on the image display means. The image alignment device according to any one of claims 1 to 4, characterized by the above.

6. The icon display means further generates an imaging icon for imaging the imaging object and an automatic adjustment icon for automatically adjusting by the automatic adjustment means after imaging the imaging object, and causes the image display means to display the same. The image alignment device according to any one of claims 1 to 4, characterized in that.

7. The image display means displays the amount of change in the stereo parameters in numerical values. The image alignment device according to any one of claims 1 to 4, characterized in that.

8. An imaging step of imaging a first image with a first camera and imaging a second image with a second camera; An automatic adjustment step of automatically adjusting and aligning the positions of the first image and the second image based on the shape information of a common imaging object in the first image and the second image; An image display step of overlapping the first image and the second image aligned by the automatic adjustment step and displaying them on an image display means; When one of the first image and the second image is a fixed image and the other is a movable image, an icon display step of generating an icon for manual operation for moving the movable image and displaying it on the image display means to prompt manual operation by an operator; A parameter change step of changing stereo parameters based on the operation amount of the manual operation icon by manual operation; An image fine adjustment step of reducing the deviation between the first image and the second image by overlapping the movable image moved in reflection of the change result of the stereo parameters and the fixed image in real time and displaying them on the image display means An image alignment method characterized by including.

Images