Image processing device and program

Abstract

Provided are an image processing apparatus and a program that can easily identify the relative positions of the imaging regions of two cameras. 【Solution means】Imaging time t m is used as the t-axis, and the position X of the moving object in the X direction A_m , X B_m , X C_m is used as the X-axis to set a t-X coordinate system. The X-axis coincides with the x A axis of the xy coordinate system set in the captured image A, and is offset by a distance d B , d C in the X direction from the x BX , d CX axis of the xy coordinate systems set in the captured images B and C. When plotting the points of each group of n1, n2, and n3 points in the t-X coordinate system, the distances d BX and d CX where the points are arranged on a straight line are identified. By identifying the distances d BX and d CX , the relative positions of the imaging regions of the infrared cameras 90B and 9OC with respect to the imaging region of the infrared camera 90A in the X direction are identified. Similarly, the identifying unit identifies the relative positions of the imaging regions of the infrared cameras 90B and 9OC with respect to the imaging region of the infrared camera 90A in the Y direction by identifying the distances d BY and d CY .

Description

[Technical Field]

[0001] This invention relates to an image processing device and a program. [Background technology]

[0002] In recent years, human detection technology using camera images has been used in a wide range of fields, including security, autonomous driving, and facility management. There are various methods for human detection, such as convolutional neural networks that automatically extract features from images, edge detection that finds patterns from shapes, background subtraction that detects moving objects by their changes from a stationary background, and infrared cameras that detect human body temperature. These human detection technologies are proving useful in a wide range of applications, including detecting suspicious individuals, monitoring the safety of residents, and autonomous driving through pedestrian detection.

[0003] Patent Document 1 is an example of a document disclosing the above technology. Patent Document 1 discloses a technology for tracking human movement by installing a thermopile array sensor, which is an infrared camera, on the ceiling and detecting and monitoring thermal images, which are camera images, at a constant frame rate. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2016-70756 [Overview of the project] [Problems that the invention aims to solve]

[0005] When tracking a person within a designated area, such as a room, using images captured by cameras (including thermal images), it is necessary to combine multiple images from multiple cameras located in different places in order to track the entire area. This process of combining multiple images requires determining the relative position of each camera's shooting area. Currently, this determination is done manually, which presents challenges such as being time-consuming and burdensome for engineers.

[0006] The present invention aims to enable easy identification of the relative positions of the shooting areas of a first camera and a second camera among a plurality of cameras. [Means for solving the problem]

[0007] The processing apparatus according to this invention comprises: an acquisition unit that acquires a plurality of captured images obtained by photographing a moving body moving linearly at a constant speed with a plurality of cameras having fixed and different shooting areas; a plurality of shooting times that each indicate the shooting timing of the plurality of captured images on a common time axis; and a specification unit that identifies the relative position of the second shooting area of ​​the second camera with respect to the first shooting area of ​​the first camera among the plurality of cameras based on the plurality of captured images and the plurality of shooting times, wherein the plurality of captured images include N first captured images obtained by photographing N times with the first camera and M second captured images obtained by photographing M times with the second camera (wherein N is an integer of 2 or more and M is an integer of 2 or more), and the specification unit includes the N first captured images and the N shooting times that each indicate the shooting timing of the N first captured images. Based on the first shooting time, N pairs of the first position and first shooting time of the moving body on the first captured image are identified, and based on the M second captured images and M second shooting times, each representing the shooting timing of the M second captured images from the plurality of shooting times, M pairs of the second position and second shooting time of the moving body on the second captured image are identified, the first position is indicated by the first x coordinate of the first xy coordinate system set in the first captured image, the second position is indicated by the second x coordinate of the second xy coordinate system set in the second captured image, the first xy coordinate system and the second xy coordinate system are set in a orientation where the x axes and y axes extend in the same direction in real space, the identification unit sets the shooting time as the t axis, the position of the moving body in the X direction coincides with the x axis of the first xy coordinate system, and the distance d in the X direction is between the x axis of the second xy coordinate system and the X direction. X When the points of the N sets and the M sets are plotted on a tX coordinate system with the X-axis shifted by a certain amount, the distance d at which each point lies on a straight line is... X By identifying this, the relative position of the second imaging region in the X direction with respect to the first imaging region is determined.

[0008] Furthermore, the program according to this invention causes a computer to function as the above-mentioned processing unit. [Effects of the Invention]

[0009] According to the present invention, the relative positions of the shooting areas of the first camera and the second camera among a plurality of cameras can be easily determined. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows the configuration of an image processing apparatus according to one embodiment of this invention. [Figure 2] This figure shows the detection positions A0 to A2 of the moving object in captured image A. [Figure 3] This figure shows the detection positions B0 to B3 of the moving object in captured image B. [Figure 4] This figure shows the detection positions C0 to C2 of the moving object in the captured image C. [Figure 5] This figure plots the detected positions in the tX coordinate system. [Figure 6] This figure plots the detection locations in the tX coordinate system, showing the detection locations aligned on a straight line. [Figure 7] This figure shows the relationship between a point on the line in the tX coordinate system and the actual detected location. [Figure 8] This figure plots the detection locations in the tY coordinate system, arranging the detection locations along a straight line. [Figure 9] This figure shows the positional relationship between captured images A to C (each shooting area of ​​the infrared camera) as identified by this embodiment. [Figure 10] This figure shows the configuration of the image processing device shown in Figure 1. [Figure 11] This figure shows another example of the positional relationship between multiple captured images (each shooting area of ​​the infrared camera) that could be identified by this embodiment. [Modes for carrying out the invention]

[0011] The embodiments of this invention will be described in detail below with reference to the drawings.

[0012] (Overview of the image processing device 10) As shown in Figure 1, the image processing device 10 according to this embodiment is connected to a plurality of infrared cameras 90A to 90C, each installed on the ceiling 81 of an area 80 such as a room. For ease of understanding, in this embodiment the number of infrared cameras is set to three, infrared cameras 90A to 90C, but the number of infrared cameras can be any number.

[0013] Multiple infrared cameras 90A to 90C are configured to capture thermal images, for example, from a thermopile array sensor. Each of the infrared cameras 90A to 90C is positioned to capture different areas of area 80 from the ceiling 81, i.e., from above. The respective areas of capture may partially overlap. The respective areas of each infrared camera 90A to 90C are fixed. Each of the infrared cameras 90A to 90C acquires a thermal image by capturing an image and also acquires the capture time, which indicates the timing of the capture on a common time axis. To acquire such a capture time, each of the infrared cameras 90A to 90C has a clock function that is synchronized to show the same time at the same timing. Each of the infrared cameras 90A to 90C supplies the captured thermal image and the capture time of that thermal image to the image processing device 10. The clock function may be provided on the image processing device 10 side. In other words, the image processing device 10 may have infrared cameras 90A to 90C capture an image to acquire a thermal image and acquire the timing of the acquisition of the thermal image as the capture time of this thermal image.

[0014] The image processing device 10 identifies the relative positional relationship of thermal images, that is, the relative positional relationship in real space of each captured area in each thermal image, based on the pair of thermal images and the time of capture from the infrared cameras 90A to 90C. The image processing device 10 arranges the thermal images according to the identified relative positional relationship to generate an overall image that allows a view of the entire area 80, and based on the generated overall image, it tracks the movement of people across the entire area 80. This eliminates the need for precise adjustment of the relative positions of the infrared cameras 90A to 90C.

[0015] (Summary of the embodiment) Before describing the details of the image processing device 10, the following outlines the method for determining the relative positional relationships of thermal images.

[0016] First, as shown in Figure 1, the mobile object 100 is moved in a straight line at a constant speed from a first position to a second position within area 80 (see arrow 101). In this example, the mobile object 100 is a person, but it may also consist of a device whose presence can be detected by infrared cameras 90A to 90C. Examples of such a mobile object 100 include a drone equipped with a heat-generating element that can be detected by thermal imaging, and a self-propelled device (e.g., a radio-controlled car). While the mobile object 100 is moving in a straight line, imaging is performed by infrared cameras 90A to 90C at predetermined intervals (which may be a fixed or variable interval). Hereinafter, the thermal image captured by infrared camera 90A will be referred to as image A. Similarly, the thermal image captured by infrared camera 90B will be referred to as image B, and the thermal image captured by infrared camera 90C will be referred to as image C. Depending on the relative distance between infrared cameras 90A to 90C (i.e., whether or not their imaging ranges overlap), the mobile object 100 may be captured by multiple infrared cameras simultaneously, or the mobile object 100 may not be captured by any camera in the nth imaging attempt. As a concrete example of the latter, the mobile object 100 may appear only in infrared camera 90A during the (n-1)th exposure, but not in any of the infrared cameras 90A-90C during the next (n)th exposure, and only appear in infrared camera 90B during the (n+1)th exposure.

[0017] Each image A, captured at predetermined intervals, is assigned a timestamp as described above. Each image A is then processed. If a moving object 100 is captured in image A, the moving object 100 is detected. The position of the detected moving object 100 (hereinafter also referred to as the detected position) is stored along with the timestamp of the image A. This detection and storage is also performed for images B and C.

[0018] Figure 2 shows examples of each detection position of the moving object 100 detected in image A. As shown in Figure 2, the time of capture t i , t i+1 , t i+2Assume that the moving object 100 is detected at each of the times. The detection position of the moving object 100 at each time is indicated by the xy coordinates in the xy coordinate system set in Image A. The origin (0, 0) of this xy coordinate system is the lower left corner of Image A. Also, the x-axis of this xy coordinate system is referred to as the x A axis, and the Y-axis is referred to as the y A axis. As shown in FIG. 2, let the detection position A0 of the moving object 100 at time t i be (x A_i , y A_i ), let the detection position A1 of the moving object 100 at time t i+1 be (x A_i+1 , y A_i+1 ), and let the detection position A2 of the moving object 100 at time t i+2 be (x A_i+2 , y A_i+2 ).

[0019] For each t i , t i+1 , t i+2 at the shooting time, each subscript indicates the number of times of shooting at that time. For example, time t i+1 is the time of the (i + 1)-th shooting after starting shooting at the above-mentioned predetermined period. Time t i , t i+1 , t i+2 may be any time among the shooting times when the moving object 100 is detected in Image A. These are the same for the following t j , t k , etc. (read i as j or k).

[0020] Examples of the detection positions of the moving object 100 detected in Image B are shown in FIG. 3. As shown in FIG. 3, assume that the moving object 100 is detected at each of the times t j , t j+1 , t j+2 , t j+3 . The detection position of the moving object 100 at each time is indicated by the xy coordinates in the xy coordinate system set in Image B. The origin (0, 0) of this xy coordinate system is the lower left corner of Image B. Also, the x-axis of this xy coordinate system is referred to as the x B axis, and the Y-axis is referred to as the y B axis. As shown in FIG. 3, at time t <00The detection position B0 of the moving object 100 is (x B_j ,y B_j ) and time t j+1 The detection position B1 of the moving object 100 is (x B_j+1 ,y B_j+1 ) and time t j+2 The detection position B2 of the moving object 100 is (x B_j+2 ,y B_j+2 ) and time t j+3 The detection position B3 of the moving object 100 is (x B_j+3 ,y B_j+3 ) Let's assume time t j , t j+1 , t j+2 , t j+3 Each time point is t i+2 It is a later time than that.

[0021] Figure 4 shows examples of each detection position of the moving object 100 detected in image C. As shown in Figure 4, the shooting time t k , t k+1 , t k+2 Assume that the moving object 100 is detected at each time point. The detection position of the moving object 100 at each time point is shown by the xy coordinates of the xy coordinate system set in image C. The origin (0,0) of this xy coordinate system is the lower left corner of image C. Also, the x-axis of this xy coordinate system is x C Axis, Y axis to y C Also called the axis. As shown in Figure 4, time t k The detection position C0 of the moving object 100 is (x C_k ,y C_k ) and time t k+1 The detection position C1 of the moving object 100 is (x C_k+1 ,y C_k+1 ) and time t k+2 The detection position C2 of the moving object 100 is (x C_k+2 ,y C_k+2 ) Also, t k , t k+1 , t k+2 Each time point is t j+3 It is a later time than that.

[0022] Note that the above x A axis, x B axis, x CThe axes are parallel to each other in real space and parallel to the X direction set on the floor of area 80 (the horizontal plane orthogonal to the optical axis directions of infrared cameras 90A to 90C). The above y A axis, y B axis, y C axis are also parallel to each other in real space and parallel to the Y direction orthogonal to the above-mentioned X direction set on the floor of area 80. x A axis, x B axis, x C axis is perpendicular to the y A axis, y B axis, y C axis respectively. 0000311

[0023] Using each set of the obtained shooting time and the xy coordinates of each detection position, calculate the respective positions of detection positions A0 to A2, B0 to B3, and C0 to C2 on the X axis commonly set in images A to C. First, extract the x coordinates among the xy coordinates of detection positions A0 to A2, B0 to B3, and C0 to C2 respectively. Then, as shown in FIG. 5, plot each point of detection positions A0 to A2, B0 to B3, and C0 to C2 in a t-X coordinate system with the horizontal axis being the shooting time t and the vertical axis being the X coordinate (the position in the X direction in area 80). The origin of the t axis in this coordinate system is set as an arbitrary time before the shooting time t i . Note that the shooting time t i , t j , t k and other times may also be regarded as the elapsed time from a common reference time. In this case, the origin of the t axis may be set as t = 0. Also, the X axis extends parallel to the X direction and coincides with the x A axis of the above image A. Thereby, the origin of the X axis coincides with the origin O A of the x A axis. The x B axis (and its origin O B ) of image B serving as the reference for detection positions B0 to B3 is set at a position shifted by a distance d XB in the X axis direction (i.e., the X direction) with respect to the X axis (and its origin). Thereby, for example, the X coordinate of point B2 is x B,j+2 +d XBとなる。 The x C axis (and its origin O of image C serving as the reference for detection positions C0 to C2B ) is set to a position shifted by a distance d in the X-axis direction (i.e., the X direction) with respect to the X-axis (and its origin). Thus, for example, the X coordinate of the detection position C1 is x XC + d C,k+1 + d XC becomes

[0024] Note that the positional relationships of the detection positions A0 to A2, B0 to B3, and C0 to C2 shown in FIG. 5, that is, the distances d XB and d XC are provisional. Since the moving object 100 to be detected moves at a constant speed, the detection positions A0 to A2, B0 to B3, and C0 to C2 will, precisely speaking, be arranged on a straight line (including a substantially straight line; that is, some deviation from the straight line of the detection positions is allowed), as shown in FIG. 6. This is because the detection positions A0 to A2, B0 to B3, and C0 to C2 plotted in the t-X coordinate system represent the time change of the position of the moving object 100 in the X direction. Therefore, the above d XB and d XC are obtained for the detection positions A0 to A2, B0 to B3, and C0 to C2 to be arranged in a straight line in the t-X coordinate system. This straight line is also referred to as straight line L1. The obtained d XB and d XC indicate the relative distances in the X-axis direction of image B and C (in other words, the imaging areas of infrared cameras 90B and 90C in area 80) with respect to image A (in other words, the imaging area of infrared camera 90A in area 80). Hereinafter, the method for obtaining d XB and d XC is shown.

[0025] First, as shown in FIG. 7, the t coordinate of the imaging time on the straight line L1 in the t-X coordinate system is set to t = t m (where m = i, i + 1, i + 2, j, j + 1, j + 2, j + 3, k, k + 1, k + 2). Also, the X coordinate of the point on the straight line L1 where t = t m is set to X m1 . That is, this coordinate is (t m , X m1 ). When the slope of the straight line L1 is a and the intercept is b, the straight line L1 satisfies the following equation (1). Further, the X coordinate when t = t m of the above detection position is Xm (For example, when m = j + 1, X j+1 =x B,j+1 +d XB ) Let's assume X m1 and X m distance e m This is obtained by the following equation (2).

number

[0026] This distance e m Regarding the above detection positions A0~A2, B0~B3, and C0~C2, the e at each position (m=i,i+1,i+2,j,j+1,j+2,j+3,k,k+1,k+2) m 2 Sum of values X The smallest possible size, d XB and d XC We will find e. m 2 This is expressed by the following equation (3).

number

[0027] Here, from the definition of the tX coordinate system, X m This is expressed by the following equation (4), S X This is expressed by the following equation (5).

number

[0028] S X These are independent variables a, b, and d. XB d XC Since it is a function that is convex upwards, S X a, b, d when is minimized XB d XC The conditions that must be satisfied are, S X a, b, d XB d XC The objective is to satisfy the following equation obtained by partially differentiating with respect to the function.

number

[0029] Let n1, n2, and n3 be the number of detection positions of the moving object 100 in images A, B, and C. Here, n1=3, n2=4, and n3=3. Based on equations (5) and (8), the following equations (10) and (11) are obtained. Also, the average value (AVG_t) of the coordinates (t,x) of the detection positions B0 to B3 of the moving object 100 in image B is given by B ,AVG_x B ) is obtained by the following equations (12) and (13).

number

[0030] From (10) to (13) above, d XB The derivation equation (14) is derived.

number

[0031] Similarly, equations (15) to (18) below can be derived from equations (5) and (9), and thus d XC The derivation equation (19) is derived.

number

[0032] Next, we find a and b. From equations (5) and (6), we obtain equation (20), and by rearranging equation (20), we obtain equation (21).

number

[0033] Equation (21) can be expressed as equation (22) below.

number

[0034] Next, equations (5) and (7) yield equation (23), and by rearranging equation (23), we obtain equation (24), and by further rearranging, we obtain equation (25).

number

[0035] Substituting equations (14) and (19) into equation (25), we obtain equations (26) and (27) below.

number

[0036] Substituting equations (27), (14), and (19) into equation (22), the equation can be transformed as shown below, ultimately yielding equation (28), from which a can be found.

number

[0037] b can be found using equations (27) and (28) above.

number

[0038] From the above, the displacement d represents the relative position of images B and C in the X-axis direction with respect to image A, as shown below. XB and d XC This can be derived.

number

[0039] Similarly, for the y-axis, a tY coordinate system is set, and the detected positions A0~A2, B0~B3, and C0~C2 are aligned on a straight line in the tY coordinate system. The distance d represents the relative position of images B and C in the Y-axis direction with respect to image A. YB and d YCWe derive the following. The derivation method is the same as the derivation method for the x-axis described above. That is, the x-axis and X-axis become the y-axis and Y-axis. The Y-axis extends parallel to the Y direction. Also, let the slope of the line L2 be f and the intercept be g. Distance d YB and d YC The derivation of the formula is shown below.

number

[0040] As shown in Figure 9, each of the above distances d XB d YB d XC d YC This determines the relative positional relationship between images A and C in a common XY coordinate system with the lower left corner of image A as the origin. This determines the positions of images B and C relative to image A. This determines the actual relative positions of the shooting ranges of infrared cameras 90A and 90C. By arranging (for example, pasting) images A and C according to the determined positional relationship, the arrangement of images A and C that reflects their relative positions in real space in area 80 (the arrangement in the overall image if the entire area 80 were photographed) is realized.

[0041] (Configuration of Image Processing Unit 10) As shown in Figure 1, the image processing unit 10 is composed of various computers, such as a personal computer. The image processing unit 10 includes a processor 11, such as a CPU (Central Processing Unit), a storage device 12, which is a non-volatile storage device that stores programs 12P executed or used by the processor 11 and various data, and a main memory 13, such as RAM (Random Access Memory), which provides a workspace for the processor 11. The programs 12P should be stored in a non-temporary medium that can be read by a computer, such as the storage device 12. The image processing unit 10 further includes an operating device 14 into which user operations are input, a display device 15 that displays various images, and I / O (Input / Output) 16, such as a communication interface connected to infrared cameras 90A to 90C.

[0042] The processor 11 operates as the acquisition unit 11A, identification unit 11B, image generation unit 11C, and tracking unit 11D, as shown in Figure 9, by executing the program 12P.

[0043] The acquisition unit 11A operates, for example, when an operation to instruct the user to start the process of determining the relative position is input to the operating device 14. After operation starts, the acquisition unit 11A controls the infrared cameras 90A to 90C to photograph the moving object 100 moving in a straight line at a constant speed. As a result, the acquisition unit 11A acquires multiple captured images A to C obtained by the infrared cameras 90A to 90C. In particular, the acquisition unit 11A causes each of the infrared cameras 90A to 90C to take multiple shots. As a result, multiple captured images A, multiple captured images B, and multiple captured images C are acquired. Furthermore, the acquisition unit 11A, using the infrared cameras 90A to 90C and its own built-in clock function, obtains multiple shooting time t that indicate the shooting timing of each of the multiple captured images A to C on a common time axis. m The function (m=i,i+1,i+2,j,j+1,j+2,j+3,k,k+1,k+2) is obtained. The shooting time may be a specific time, or it may be the elapsed time from any point before shooting by infrared cameras 90A~90C begins.

[0044] The multiple captured images A to C acquired by the acquisition unit 11A may, in particular, be images in which the moving object 100 is visible among all captured images. The acquisition unit 11A performs a detection process for the moving object 100 on all captured images from the infrared cameras 90A to 90C, and acquires multiple captured images in which the moving object 100 was detected as multiple captured images A to C.

[0045] Multiple captured images A to C include n1 captured images A obtained by taking n1 shots (3 shots in the above example, but any number of shots is acceptable) with infrared camera 90A, n2 captured images B obtained by taking n2 shots (4 shots in the above example, but any number of shots is acceptable) with infrared camera 90B, and n3 captured images C obtained by taking n3 shots (3 shots in the above example, but any number of shots is acceptable) with infrared camera 90C.

[0046] The specific unit 11B contains multiple captured images A to C and multiple capture time t. m Based on this, the relative positions of the imaging areas of infrared cameras 90B and 90C with respect to the imaging area of ​​infrared camera 90A are determined.

[0047] More specifically, the specific unit 11B contains n1 captured images A and n1 capture times t that indicate the capture timing of each of the n1 captured images A. m Based on (m=i,i+1,i+2), the position of the moving object 100 in the X direction on the captured image A (the x coordinate is x A_m ) and the time t of the captured image A at that position m n1 pairs (t i ,x A_i ),(t i+1 ,x A_i+1 ),(t i+2 ,x A_i+2 The identification unit 11B identifies the x and y coordinates of the moving object 100 captured in the captured image A, and identifies the pair based on the identified x coordinates and the time the captured image A was taken. The same applies to the following captured images B and C.

[0048] The specific unit 11B contains n2 captured images B and n2 shooting times t, which indicate the shooting timing of each of the n2 captured images B. m Based on (m=j,j+1,j+2,j+3), the position of the moving object 100 in the X direction on the captured image B (the x coordinate of the xy coordinate system set for the captured image B is x B_m ) and the time t of the captured image B at that position m n pairs of (t j ,x B_j ),(t j+1 ,x B_j+1 ),(t j+2 ,x B_j+2 ),(t j+3 ,x B_j+3 Identify )

[0049] The specific unit 11B contains n3 captured images C and n3 shooting times t, which indicate the shooting timing of each of the n3 captured images C. m Based on (m=k,k+1,k+2), the position of the moving object 100 in the X direction on the captured image C (the x coordinate of the xy coordinate system set in the captured image C is x C_m ) and the time t of the captured image C at that position m n3 pairs (t k ,x C_k ),(t k+1 ,x C_k+1 ),(t k+2 ,x C_k+2 Identify )

[0050] The x and y coordinate systems set for each of the captured images A through C are configured such that the x axes and y axes extend in the same direction in real space (the area 80 being photographed).

[0051] The specific part 11B is the shooting time t m The position of the moving body 100 in the X direction is defined as the t-axis. A_m , X B_m , X C_m Set up a tX coordinate system with the x axis being the x of the xy coordinate system set for captured image A. A The x axis coincides with the xy coordinate system set for captured images B and C. B , x C Distance d in the X direction from the axis BX d CX The only difference is d. The specific part 11B determines the distance d at which each point lies on a straight line when the points of n1 sets, n2 sets, and n3 sets are plotted in the tX coordinate system. BX and d CX Identify the distance d (see Figure 6). BX and d CX This can be derived using equations (14) and (19) above. Distance d BX and d CX By identifying the distance d, the relative positions in the X direction (in other words, the X-axis direction) of the imaging areas of infrared cameras 90B and 90C with respect to the imaging area of ​​infrared camera 90A are determined. Similarly, the identification unit 11B determines the distance d by formulas (30) and (31) above. BY and dCY By identifying these factors, the relative positions of the imaging areas of infrared cameras 90B and 90C in the Y-direction (in other words, the Y-axis direction) with respect to the imaging area of ​​infrared camera 90A are determined. This makes it easy to determine the relative positions of the imaging areas of infrared cameras 90B and 90C with respect to the imaging area of ​​infrared camera 90A.

[0052] The image generation unit 11C sequentially generates a new image (an image overlooking area 80) by arranging the newly captured images A to C, each of which is taken sequentially by the infrared cameras 90A to 90C, so that they correspond to the relative positions identified by the identification unit 11B. This generates an image in which captured images A to C are accurately arranged (the positional relationship is accurately reflected in the actual shooting area). The image generation unit 11C may sequentially display the generated images on the display device 15. The tracking unit 11D may monitor the newly generated images and track moving objects such as people based on these images. The tracking unit 11D may monitor the entry and exit of moving objects into and out of area 80 through the tracking and display the monitoring results on the display device 15. The tracking unit 11D may also output the monitoring results to an air conditioning unit or the like. The air conditioning unit may perform air conditioning control based on the monitoring results (such as the number of people in area 80).

[0053] Using the method described above, even if the number of infrared cameras is less than three, simply by having the moving object 100 move in one direction at a constant speed when determining the relative position of the images, the positions of the images from all cameras that detected the moving object 100 can be automatically determined and the images placed. Furthermore, by repeating this process multiple times, the placement (stitching) of images from all cameras within area 80 can be automatically performed. This results in an image in which the images captured by the cameras are appropriately placed. Figure 11 shows an example of the stitched image (see the dashed rectangle). In Figure 11, the black circles indicate the detection positions of the moving object 100, and the solid lines connecting the black circles indicate the movement trajectory of the moving object 100.

[0054] The camera may be any other camera besides an infrared camera (e.g., a visible light camera, an ultraviolet camera, etc.) that can obtain images capable of detecting moving objects. It is sufficient to determine the relative positional relationship between multiple images so that the points representing the time of capture and the position in the X or Y direction lie on a straight line, using a tX coordinate system and tY coordinate system commonly set across the multiple images. The method of determination is not limited to the above. For example, an artificial intelligence performing image recognition may move the images in the tX coordinate system or tY coordinate system, in which each point is plotted, so that each point lies on a straight line, and determine the relative positional relationship between each image from the positional relationship of each image.

[0055] Multiple cameras may be arranged in a single line along the X-axis, rather than in the matrix shown in Figure 11. In such cases, the distance d in the tY coordinate system is... YB d YC The calculation becomes unnecessary.

[0056] (Scope of the present invention) The present invention is not limited to the embodiments described above. For example, the present invention includes various modifications to the above embodiments and modifications that can be understood by those skilled in the art within the scope of the technical concept of the present invention. For example, at least a part of each of the above parts 11A to 11D may be composed of an FPGA, ASIC, etc. The configurations listed in the above embodiments and modifications can be combined as appropriate within a non-contradictory range. It is also possible to delete any of the above configurations.

[0057] (Note) The following examples illustrate configurations based on the above embodiments and modifications. Any partial configuration of the above embodiments and modifications may be applied to each appendix. Furthermore, parts of each appendix may be combined. (Note 1) An acquisition unit that acquires multiple captured images obtained by photographing a moving object moving in a straight line at a constant speed with multiple cameras that have fixed and different shooting areas, and multiple shooting times that indicate the shooting timing of each of the multiple captured images on a common time axis, The system includes a specification unit that identifies the relative position of the second shooting area of ​​the second camera to the first shooting area of ​​the first camera among the plurality of cameras based on the plurality of captured images and the plurality of shooting times, The plurality of captured images include N first captured images obtained by taking N shots with the first camera and M second captured images obtained by taking M shots with the second camera (wherein N is an integer of 2 or more, and M is an integer of 2 or more), The specified part is, Based on the N first captured images and the N first shooting times, each representing a different shooting timing for the N first captured images, N pairs of the first position of the moving object on the first captured image and the first shooting time are identified. Based on the M second images and the M second shooting times, each representing a shooting timing of one of the M second images, the M pairs of the second position of the moving object on the second images and the second shooting time are identified. The first position is indicated by the first x-coordinate of the first xy-coordinate system set in the first captured image. The second position is indicated by the second x-coordinate of the second xy-coordinate system set in the second captured image. The first xy coordinate system and the second xy coordinate system are set up so that the x axes and y axes extend in the same direction in real space. The identifying unit identifies the relative position of the second shooting region in the X direction with respect to the first shooting region by identifying the distance dX at which each of the N sets and the M sets of points lies when plotted in the tX coordinate system, where the shooting time is the t-axis, the position of the moving body in the X direction coincides with the x-axis of the first xy coordinate system, and the x-axis is shifted by a distance dX in the X direction from the x-axis of the second xy coordinate system. Image processing device. (Note 2) The first position is indicated by the first x coordinate and the first y coordinate of the first xy coordinate system. The second position is indicated by the second x coordinate and the second y coordinate of the second xy coordinate system, The identifying unit identifies the relative position of the second imaging region in the Y direction with respect to the first imaging region by identifying the distance dy at which each of the N sets and M sets of points lies on a straight line when plotting each of the points on the tX coordinate system, where the shooting time is the t-axis, the position of the moving body in the Y direction coincides with the y-axis of the first xy coordinate system, and the Y-axis is shifted by a distance dY in the Y direction from the y-axis of the second xy coordinate system. The image processing device described in Appendix 1. (Note 3) The system further includes an image generation unit that generates an image in which an image captured by the first camera and an image captured by the second camera are arranged so as to be in the positional relationship of the specified relative positions. The image processing apparatus described in Appendix 1 or 2. (Note 4) A program that causes a computer to function as an image processing device as described in one of the appendices 1 to 3. [Explanation of Symbols]

[0058] 10...Image processing device, 11...Processor, 11A...Acquisition unit, 11B...Specification unit, 11C...Image generation unit, 11D...Tracking unit, 12...Storage, 12P...Program, 13...Main memory, 14...Operation device, 15...Display device, 80...Area, 81...Ceiling, 90A~90C...Infrared camera, 100...Moving object, 101...Movement trajectory, A~C...Captured images.

Claims

1. An acquisition unit acquires multiple captured images obtained by photographing a moving object moving in a straight line at a constant speed with multiple cameras that have fixed and different shooting areas, and multiple shooting times that indicate the shooting timing of each of the multiple captured images on a common time axis. The system includes a specification unit that identifies the relative position of the second shooting area of ​​the second camera with respect to the first shooting area of ​​the first camera among the plurality of cameras, based on the plurality of captured images and the plurality of shooting times, The plurality of captured images include N first captured images obtained by taking N shots with the first camera and M second captured images obtained by taking M shots with the second camera (wherein N is an integer of 2 or more, and M is an integer of 2 or more). The specified part is, Based on the N first captured images and the N first shooting times, each representing a shooting timing for one of the N first captured images, N pairs of the first position of the moving object on the first captured image and the first shooting time are identified. Based on the M second images and the M second shooting times, each representing a shooting timing of one of the M second images, M pairs of the second position of the moving object on the second images and the second shooting time are identified. The first position is indicated by the first x-coordinate of the first xy-coordinate system set in the first captured image, The second position is indicated by the second x-coordinate of the second xy-coordinate system set in the second captured image, The first xy coordinate system and the second xy coordinate system are set up so that the x axes and y axes extend in the same direction in real space. The specified unit uses the shooting time as the t-axis, and the position of the moving body in the X direction coincides with the x-axis of the first xy coordinate system, and the distance d in the X direction from the x-axis of the second xy coordinate system. Ｘ When the points of the N sets and the M sets are plotted on a t-X coordinate system with the X-axis shifted by a certain amount, the distance d at which each point lies on a straight line is... Ｘ By identifying the X-direction relative position of the second imaging region with respect to the first imaging region, Image processing device.

2. The first position is indicated by the first x coordinate and the first y coordinate of the first xy coordinate system, The second position is indicated by the second x coordinate and the second y coordinate of the second xy coordinate system, The specified unit uses the shooting time as the t-axis, and the position of the moving body in the Y-direction coincides with the y-axis of the first xy coordinate system, and the distance d in the Y-direction from the y-axis of the second xy coordinate system. Ｙ When the points of the N sets and the M sets are plotted on a t-X coordinate system with the Y-axis shifted by a certain amount, the distance d at which each point lies on a straight line is... ｙ By identifying the first imaging area, the relative position of the second imaging area in the Y direction is determined. The image processing apparatus according to claim 1.

3. The system further includes an image generation unit that generates an image in which an image captured by the first camera and an image captured by the second camera are arranged to have a positional relationship of the specified relative positions. The image processing apparatus according to claim 1.

4. A program that causes a computer to function as an image processing device according to claim 1.

Images