Image processing device, image processing system, image processing method and program therefor

Abstract

To achieve camera calibration under fewer constraint conditions.SOLUTION: An image processing device calculates an interframe projective transformation matrix, specifies an image correspondence point corresponding four or more real spatial coordinate known points on a real space plane with an arbitrary frame of a frame group composed of a plurality of frames, aggregates the image correspondence point in a reference frame on the basis of the interface projective transformation matrix, calculates a real space projective transformation matrix of the reference frame on the basis of the real space coordinate known points and the image correspondence point, calculates a real space projective transformation matrix of another frame on the basis of the interframe projective transformation matrix between the reference frame and an image of the other frame, determines an image correspondence point in the other frame on the basis of the interframe projective transformation matrix, and adjusts at least a portion of a camera parameter so as to reduce difference between an estimated value of an image correspondence point corresponding to a real space coordinate known point estimated from the camera parameter constituting the real space projective transformation matrix and the image correspondence point corresponding to the real space coordinate known point.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present application relates to an image processing device, an image processing system, an image processing method, and a program therefor, and in particular to camera calibration of a captured image. [Background technology]

[0002] Camera calibration is the process of acquiring the shooting-related parameters necessary for analyzing captured images. These shooting-related parameters are also called camera parameters. Camera parameters include intrinsic and extrinsic parameters. Intrinsic parameters include, for example, the focal length of the lens. Extrinsic parameters include the position and orientation of the camera. Camera calibration can also be considered the process of identifying a projective transformation matrix constructed based on the camera parameters. The projective transformation matrix represents the relationship between the coordinates of an object in the real-space coordinate system and the coordinates of the object as they appear in the camera image coordinate system. The real-space coordinate system is a coordinate system that provides two-dimensional or three-dimensional coordinates indicating any position in real space. The camera image coordinate system is a coordinate system that provides two-dimensional coordinates indicating any position in the captured image captured by the camera. In general, the success and accuracy of video analysis constructed from captured images are affected by camera calibration. Therefore, accurate camera calibration is extremely important in video analysis.

[0003] In conventional camera calibration, the position, orientation, and zoom of the camera are fixed, and a calibration image is captured in advance without changing the angle of view during capture, and a projective transformation matrix is ​​calculated using the calibration image. The calculated projective transformation matrix is ​​then used to analyze a separate image captured thereafter. When capturing an image for calibration, a predetermined point with known real-space coordinates (referred to herein as a "known real-space coordinate point") is included in the camera's field of view. The projective transformation matrix is ​​calculated using the relationship between the camera image coordinates of the known real-space coordinate point that appears in the captured image and the real-space coordinates in the real space.

[0004] The number of known real space coordinate points used to calculate the projective transformation matrix varies depending on the camera calibration method. For example, in the DLT (Direct Linear Transformation) method described in Non-Patent Document 1, the minimum number of known points of real space coordinates required is six. Patent Document 1 proposes a method of using three known points of real space coordinates by searching for camera internal parameter values ​​using an optimization method. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-116280 [Non-patent literature]

[0006] [Non-Patent Document 1] Y. I Abdel-Aziz, HM Karara, Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. ASP symposium on close-range photogrammetry, Falls Church, VA, 1-19, 1971. [Non-patent document 2] Junji Takamatsu, Michiyoshi Ae, Norihisa Fujii, Development of the Panning DLT Method for a Large Measurement Range, Journal of Physical Education, 42: 19-29, 1997 Summary of the Invention [Problem to be solved by the invention]

[0007] Fixing the camera's angle of view limits the range of real space that appears in the captured image. Therefore, methods have been proposed for analyzing images captured while changing the angle of view. For example, in the panning DLT method described in Non-Patent Document 2, the variable angle of view is limited to pan and tilt, and images are captured while the camera position is fixed using a tripod. Then, in images captured so that a reference point whose real-space coordinates do not change in real space is always included in the field of view, a corresponding point is identified for each frame, and camera calibration is performed using the camera image coordinates of the identified point. However, the method described in Non-Patent Document 2 requires that the reference point appear in the captured image in each frame. Furthermore, this method cannot accommodate changes in the field of view that accompany changes in the angle of view due to rotation, zoom, etc.

[0008] The present application has been made in consideration of the above points, and aims to provide an image processing device, an image processing system, an image processing method, and a program therefor that can realize camera calibration under fewer constraints. [Means for solving the problem]

[0009] (1) The present application has been made to solve the above-mentioned problems, and one aspect of the present application includes calculating an inter-frame projective transformation matrix that indicates a transition of images between frames, identifying image corresponding points that correspond to four or more points with known real-space coordinates, which are points with known real-space coordinates on a real-space plane, in any frame of a frame group consisting of multiple frames, aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculating a real-space projective transformation matrix for the reference frame based on the image corresponding points aggregated with the known real-space coordinate points, calculating a real-space projective transformation matrix for the other frame based on the inter-frame projective transformation matrix between the image of the reference frame and the image of the other frame and the real-space projective transformation matrix, determining image corresponding points in the other frame based on the inter-frame projective transformation matrix, and adjusting at least a portion of the camera parameters for each frame so as to reduce a difference between an estimated value of the image corresponding point corresponding to the known real-space coordinate point, which is estimated based on camera parameters that constitute the real-space projective transformation matrix, and the image corresponding point corresponding to the known real-space coordinate point.

[0010] (2) Another aspect of the present application is a method for an image processing device, the method comprising: calculating an inter-frame projective transformation matrix that indicates a transition of images between frames; identifying image corresponding points that correspond to four or more known real-space coordinate points, which are points whose real-space coordinates are known on a real-space plane, in any frame of a frame group consisting of multiple frames; aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix; calculating a real-space projective transformation matrix for the reference frame based on the image corresponding points aggregated with the known real-space coordinate points; calculating a real-space projective transformation matrix for the other frame based on the inter-frame projective transformation matrix between the image of the reference frame and the image of the other frame and the real-space projective transformation matrix; determining image corresponding points in the other frame based on the inter-frame projective transformation matrix; and adjusting at least a portion of the camera parameters for each frame so as to reduce a difference between an estimated value of the image corresponding point corresponding to the known real-space coordinate point, which is estimated based on camera parameters that constitute the real-space projective transformation matrix, and the image corresponding point corresponding to the known real-space coordinate point. [Effects of the Invention]

[0011] According to the present invention, camera calibration can be achieved under fewer constraints, which facilitates image analysis in a wider range of shooting conditions and under a variety of shooting conditions. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a schematic block diagram illustrating an example of the functional configuration of an image processing system according to an embodiment of the present invention. [Figure 3] FIG. 1 is an explanatory diagram illustrating a projective transformation. [Figure 4] FIG. 1 is an explanatory diagram illustrating inter-frame projective transformation. [Figure 5] FIG. 1 is a diagram illustrating an example of optical flow. [Figure 6]FIG. 1 is a first explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 7] FIG. 10 is a second explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 8] FIG. 10 is a third explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 9] FIG. 4 is a fourth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 10] FIG. 5 is a fifth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 11] FIG. 6 is a sixth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 12] FIG. 7 is a seventh explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 13] FIG. 8 is an eighth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 14] FIG. 9 is a ninth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 15] FIG. 1 is an explanatory diagram illustrating an example of the relationship between inter-frame projective transformation and real space inter-image projective transformation. [Figure 16] FIG. 1 is an explanatory diagram illustrating bundle adjustment. [Figure 17] FIG. 10 is an explanatory diagram illustrating an example of the relationship between the position of a camera and the focal length error. [Figure 18] 1 is a flowchart illustrating an image processing method according to the present embodiment. [Figure 19] 10 is a flowchart showing a first example of a process for calculating an inter-frame projective transformation matrix. [Figure 20] 10 is a flowchart showing a second example of the process of calculating an inter-frame projective transformation matrix. [Figure 21] 10 is a flowchart illustrating a smoothing process of an inter-frame projective transformation matrix. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. 1 is an explanatory diagram showing an overview of an image processing system S1 according to this embodiment. The image processing system S1 includes an image processing device 10 and a camera 20. The image processing device 10 acquires image data from the camera 20, representing images captured at different times for each frame (sometimes referred to herein as "captured images"). The image processing device 10 performs calibration on the image represented by the acquired image data and estimates a homography matrix for each frame based on the parameters of the camera 20 (sometimes referred to herein as "camera parameters"). The homography matrix indicates the correspondence between a coordinate system set in the three-dimensional real space of the subject of image capture (sometimes referred to herein as "real space coordinate system") and a two-dimensional coordinate system set in the captured image (sometimes referred to herein as "camera image coordinate system" or "image coordinate system"). In the example of FIG. 3, a vector m indicating coordinates (u, v) in the captured image i p corresponds to the product of a vector M indicating coordinates (X, Y, Z) in the real space, a homography matrix P, and a scale factor k. The homography matrix P is uniquely determined based on the camera parameters. The image processing device 10 analyzes the image for each frame based on the estimated homography matrix.

[0014] In the calibration according to this embodiment, the image processing device 10 calculates an inter-frame projective transformation matrix that indicates the transition of images between frames. In any frame of a frame group consisting of images of multiple frames, the image processing device 10 identifies four or more image corresponding points that correspond to points with known real-space coordinates on a real-space plane. The image processing device 10 determines one reference frame from the frame group as a reference, and determines image corresponding points in the reference frame that correspond to image corresponding points in the arbitrary frame based on an inter-frame projective transformation matrix between the arbitrary frame in which image corresponding points have been identified and the reference frame. The image processing device 10 calculates a real-space projective transformation matrix based on the correspondence between points with known real-space coordinates and image corresponding points in the reference frame. The image processing device 10 calculates a real-space projective transformation matrix for another frame based on the inter-frame projective transformation matrix between the reference frame and another frame and the real-space projective transformation matrix of the reference frame. The image processing device 10 determines image corresponding points in another frame that correspond to image corresponding points in the reference frame based on the inter-frame projective transformation matrix between the reference frame and another frame. The image processing device 10 determines, for each frame, the difference between the estimated value of the image corresponding point estimated based on the camera parameters constituting the real-space projective transformation matrix and the coordinate of the image corresponding point corresponding to the real-time coordinate known point as a reprojection error, and adjusts some or all of the camera parameters so as to minimize the reprojection error. The image processing device 10 analyzes the captured image of that frame using the camera parameters determined for that frame.

[0015] The camera 20 captures images showing the scene of a subject located within its field of view at different times. The camera 20 is, for example, a digital video camera. A digital video camera has the function of continuously capturing moving images. In general, moving images are composed of still images of different frames captured at different times, and are captured or displayed at a predetermined frame rate (for example, 30 to 300 fps). The camera 20 transmits image data showing the captured images to the image processing device 10.

[0016] The image processing system S1 realizes calibration for each frame of a captured image, allowing for changes over time in the orientation and zoom of the camera 20. However, it is necessary to set four or more feature points as known real-space coordinates on a plane that spreads over a relatively wide area in real space, and to set the coordinates of each of these points in advance in the image processing device 10. The feature points are artificial or natural objects that have morphological characteristics that can be easily distinguished from other objects or patterns.

[0017] The image processing system S1 according to this embodiment is applied to captured images obtained by continuously capturing moving subjects from a distance, for example, at a stadium or other event venue. In the example of FIG. 1, a scene of an athlete running on a track Tr in an athletics stadium is captured, and hurdle placement marks Mk on the track Tr are used as feature points for known real-space coordinate points. Feature points do not necessarily need to be captured in advance. Four or more feature points do not necessarily need to appear in all frames of the image. As long as they appear in at least some of the frames to be processed at once, only three or fewer feature points may appear in any one of the frames, or there may be frames in which no feature points appear at all. In this way, marks used in events such as competitions, or patterns or objects that do not hinder the movement of the subject and are not continuously occluded by the subject or other objects, can be used as known real-space coordinate points. Feature points may also be used as references for the real-space coordinate system. For example, a horizontal plane on which multiple feature points are arranged may be defined as a plane whose vertical (z) coordinate value is zero, or any one of these points or the center of gravity may be defined as the origin of the real space coordinate system.

[0018] The position and orientation of the camera 20 do not need to be fixed using a device such as a tripod. Changes in either or both of the position and orientation of the camera 20 are permitted. Changes in the orientation of the camera 20 can be panned, tilted, or rotated. For example, the image processing device 10 according to this embodiment may also process images captured by a photographer holding the camera 20 while moving and adjusting the zoom to follow the movement of the subject. Therefore, even in a large space with one side or diameter of the shooting range exceeding 100 m, it is possible to process images captured using zooming and other camerawork, as long as the space has known real-space coordinates.

[0019] Next, an example of the functional configuration of the image processing system S1 according to this embodiment will be described. 2 is a schematic block diagram showing an example of the functional configuration of an image processing system S1 according to this embodiment. In the example of FIG. 2, the image processing system S1 includes an image processing device 10, a camera 20, and a display 30. The image processing device 10 includes an arithmetic processing unit 120, a storage unit 140, an input / output unit 150, and an operation input unit 160. The image processing device 10 may be configured as a general-purpose information processing device such as a personal computer or a workstation, or may be configured as part of a dedicated image production device or image editing device.

[0020] The arithmetic processing unit 120 performs processing and control to realize various functions of the image processing device 10. The arithmetic processing unit 120 may be configured using dedicated components, or may be configured as a general-purpose computer system including a processor and a storage medium. The processor may read a predetermined program stored in advance in a storage medium and execute the read program to realize the functions of each functional unit in cooperation with the storage medium and other hardware. In this application, "executing a program" refers to executing processing instructed by instructions written in the program. The processor includes, for example, a CPU (Central Processing Unit). The processor may also be equipped with a different type of arithmetic processing device than the CPU, for example, a GPU (Graphics Processing Unit). Each functional unit of the arithmetic processing unit 120 will be described later.

[0021] The storage unit 140 stores data used for various processes in the arithmetic processing unit 120 and data acquired or generated by the arithmetic processing unit 120. The storage unit 140 includes an auxiliary storage device such as a hard disk drive (HDD) or a solid-state storage device (SSD). The input / output unit 150 inputs and outputs various types of data to and from other devices via a wired or wireless connection. The input / output unit 150 outputs image data input from the camera 20 to the arithmetic processing unit 120. The input / output unit 150 outputs display data input from the arithmetic processing unit 120 to the display 30. The input / output unit 150 includes, for example, an input / output interface. The input / output unit 150 may be connected to other devices via a communication network so as to be able to send and receive data thereto. The operation input unit 160 receives a user operation and generates an operation signal in accordance with the received operation. The operation input unit 160 outputs the generated operation signal to the arithmetic processing unit 120. The operation input unit 160 may have a general-purpose input device such as a mouse or a touch sensor, or may have a dedicated input device such as a button, a lever, or a dial.

[0022] The display 30 displays a display screen indicated by display data input from the image processing device 10. The display 30 may be any type of display monitor, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display.

[0023] One or both of the camera 20 and the display 30 may be configured separately from the image processing device 10, as exemplified in Fig. 2, or may be configured integrally with the image processing device 10. Furthermore, the operation input unit 160 may be configured integrally with the image processing device 10, as exemplified in Fig. 2, or may be configured separately. The touch sensor and the display 30 constituting the operation input unit 160 may be configured separately, or may be configured integrally as a touch panel.

[0024] Next, we will explain an example of the functional configuration of the arithmetic processing unit 120. The arithmetic processing unit 120 includes an image acquisition unit 122, an inter-frame analysis unit 124, a known coordinate point identification unit 126, a real space inter-image analysis unit 128, a calibration unit 130, and an output processing unit 132.

[0025] The image acquisition unit 122 receives image data input from the camera 20 using the input / output unit 150. The image acquisition unit 122 sequentially stores the acquired image data in the storage unit 140. The storage unit 140 accumulates image data acquired at different times. The inter-frame analysis unit 124 calculates an inter-frame projective transformation matrix that indicates the transition of images between frames at different times. The inter-frame analysis unit 124 reads image data to be processed from the storage unit 140 and calculates optical flow, which indicates the movement from the image of the current frame shown in the read image data to the image of another frame. The optical flow is expressed by a motion vector that indicates the amount and direction of movement for each part of the image. The optical flow is calculated so that the pattern at a certain part (starting point) in the current frame is as similar as possible to the pattern at a part (ending point) after movement in another frame.

[0026] As shown in FIG. 4, the inter-frame analysis unit 124 calculates the coordinates [u, v, 1] of a certain part in the current frame i. T (T denotes the transpose of a vector or matrix), the pattern at this coordinate is calculated by the inter-frame projection transformation matrix H i→i+1 f The corresponding coordinates after movement in another frame i+1, [u', v', 1], are obtained by multiplying T The inter-frame projection transformation matrix H i→i+1 f However, in Figure 4, h 00 ,…,h 21 ,1 is the inter-frame projective transformation matrix H i→i+1 f where k denotes an element of the inter-frame projective transformation matrix. Furthermore, k is a scale factor that expresses the arbitrariness of the constant multiple of the inter-frame projective transformation matrix. The inter-frame analysis unit 124 searches for the inter-frame projective transformation matrix by, for example, performing a procedure such as regressing the optical flow calculated between frames. The inter-frame analysis unit 124 can also calculate the inter-frame projective transformation matrix between other pairs of frames. For example, the inter-frame analysis unit 124 sequentially calculates the inter-frame projective transformation matrix from one frame to the next frame for each frame. The inter-frame analysis unit 124 stores the inter-frame projective transformation matrix calculated for each pair of frames in the storage unit 140.

[0027] When calculating the optical flow, the inter-frame analysis unit 124 can use known techniques, such as block matching, feature point correspondence, etc. According to block matching, the optical flow is calculated for each block that constitutes part of the current frame and that is made up of a predetermined number of pixels. Fig. 5 shows an example of the optical flow Of calculated by block matching. In the example of Fig. 5, the optical flow from each starting block, which is divided into regular intervals by a grid and has a plurality of pixels, to the end block is shown. Block matching searches for the end block whose pattern most closely matches the pattern of the starting block, and whose matching degree is equal to or exceeds a predetermined reference value.

[0028] In the image illustrated in FIG. 5, accurate optical flow may not be calculated in or around a non-planar region. Even in a planar region, accurate optical flow may not be calculated for a region with constant brightness or color (monotone) or a region showing a spatially repeated pattern. These optical flows containing errors can prevent accurate estimation of the inter-frame projective transformation matrix. To reduce these effects, the inter-frame analysis unit 124 may determine the inter-frame projective transformation matrix by regressing the optical flow using a robust method. The inter-frame analysis unit 124 may use, for example, the RANSAC (Random Sample Consensus) method. Specific examples of methods for calculating the inter-frame projective transformation matrix will be described later as Methods 1 to 7.

[0029] Returning to FIG. 2, the coordinate-known point identifying unit 126 identifies the portion where each of the four or more real-space coordinate-known points appears as an image-corresponding point for each of the four or more real-space coordinate-known points from the real space represented in the image (digitizes the real-coordinate-known points). The known coordinate point identification unit 126 causes the display 30 to display a display screen on which at least one frame of an image to be analyzed (which may be referred to as a "target image" in the present application) is arranged for the output processing unit 132, for example. The known coordinate point identification unit 126 specifies a part within the target image indicated by an operation signal input from the operation input unit 160 as an image corresponding point. As a result, while viewing the display screen displayed on the display 30, the user can recognize the feature points where the known coordinate points are set and indicate the part where the feature points appear by an operation. In the image of the track Tr illustrated in FIG. 1, the position of the mark Mk is indicated as the image corresponding point. Regarding the known real-space coordinate points that are the targets of specifying the image corresponding points, it is sufficient that four or more appear in any frame in the group of frames to be processed at one time. The four or more different known real-space coordinate points may all be represented in one frame, or may be dispersed and represented in two or more frames. The known coordinate point identification unit 126 stores, in the storage unit 140, known coordinate point information indicating the coordinates of the frames and the image corresponding points specified for each known real-space coordinate point.

[0030] The real-space image-to-image analysis unit 128 determines one frame in the group of frames to be processed as a reference frame. The real-space image-to-image analysis unit 128 can determine, for example, the first frame, or the central frame, or the last frame in the group of frames to be processed, or the frame in which the coordinates of the most image corresponding points for the known real-space coordinate points indicated in the known coordinate point information stored in the storage unit 140 are specified, as the reference frame. The real-space image-to-image analysis unit 128 uses the inter-frame projective transformation matrix stored in the storage unit 140 to determine the inter-frame projective transformation matrix from the frame in which the image corresponding points of the known real-space coordinate points are specified to the reference frame. The real-space image-to-image analysis unit 128 determines the inter-frame projective transformation matrix H from the frame j (j < i) preceding the reference frame i to the reference frame i as H j→i f as H i-1→i f …H j+1→j+2 f H j→j+1 fThe real space inter-image analysis unit 128 calculates an inter-frame projective transformation matrix H j→i f (H i→i+1 f ) -1 …(H j-2→j-1 f ) -1 (H j-1→j f ) -1 The real-space inter-image analysis unit 128 can determine the coordinates of the image corresponding points in the reference frame by multiplying the coordinates of the image corresponding points identified in that frame by the determined inter-frame projective transformation matrix. In this way, the image corresponding points are aggregated in the reference frame, and the coordinates of the image corresponding points in the reference frame are determined for each of the four or more real-space coordinate known points.

[0031] The real space inter-image analysis unit 128 calculates a real space projection transformation matrix H for transforming an arbitrary point M' on a plane in the real space coordinate system to a point m in the camera image coordinate system based on the real space coordinates and the image corresponding points for each real space coordinate known point in the reference frame k. k g Calculate the real space projection transformation matrix H g The calculation method will be described later in Appendix 4. The real space inter-image analysis unit 128 calculates the real space projective transformation matrix H k g The inter-frame projective transformation matrix H from the reference frame to the other frame is k→j f and multiplying it by , the real space projection transformation matrix H j g FIG. 15 shows the real space projection transformation matrix H i-1 g Then, the inter-frame projective transformation matrix H i f The matrix H obtained by multiplying i f H i-1 g is the real space projection transformation matrix H ig In addition, the real space projection transformation matrix H i g and the inter-frame projective transformation matrix H i+1 f By multiplying by , the real space projection transformation matrix H i+1 g The real space inter-image analysis unit 128 stores the real space projective transformation matrix calculated for each frame in the storage unit 140. The real space projective transformation matrix will be described later in Supplementary Note 4.

[0032] The calibration unit 130 adjusts the camera parameters (bundle adjustment) so that image corresponding points in the camera image coordinate system corresponding to points with known real-space coordinates in each frame closely match estimated values ​​of image corresponding points transformed from the known real-space coordinate points in that frame using camera parameters constituting a real-space projective transformation matrix. The camera parameters to be adjusted may be limited to some element parameters (e.g., focal length), and other element parameters may be fixed to constant values. The calibration unit 130 determines image corresponding points in each of the other frames using an inter-frame projective transformation matrix for each image corresponding point in the reference frame. The calibration unit 130 sets the camera parameters related to the real-space projective transformation matrix as initial values ​​and calculates estimated values ​​of image corresponding points estimated for points with known real-space coordinates using the camera parameters. The calibration unit 130 regards the difference between the image corresponding points and their estimated values ​​as a reprojection error and searches for camera parameters that minimize the reprojection error. 16 illustrates a case in which the difference in coordinate values ​​between an image corresponding point m and its estimated value m' is defined as a reprojection error, and the camera parameters are adjusted so as to reduce the reprojection error. The calibration unit 130 stores the camera parameters determined for each frame in the storage unit 140. The camera parameters will be described later in Supplementary Note 5. In particular, bundle adjustment will be described in Supplementary Note 5-3. The calibration unit 130 may analyze the image of each frame using camera parameters determined for each frame. The calibration unit 130 stores corrected image data representing the corrected image in the storage unit 140.

[0033] The output processing unit 132 outputs various types of data to the outside of the image processing device 10. The output processing unit 132 may configure a display screen and output display data indicating the configured display screen to the display 30, thereby displaying the display screen. The output processing unit 132 may analyze or correct the acquired image using the camera parameters obtained by the calibration unit 130. The output processing unit 132 may output display data representing the corrected image to the display 30, thereby displaying the corrected image. When the output processing unit 132 receives a corrected image request command from another device via the input / output unit 150, the output processing unit 132 may transmit corrected image data indicating the corrected image to the other device that is the sender. The output processing unit 132 may output display data representing the analysis results to the display 30, thereby displaying the analysis results. When the output processing unit 132 receives an analysis command from another device via the input / output unit 150, the output processing unit 132 may transmit analysis data indicating the analysis result to the other device that sent the command or to another device.

[0034] Next, an example of an image processing method according to this embodiment will be described below with reference to a flowchart shown in FIG. (Step S102) The image acquisition unit 122 of the image processing device 10 receives image data representing a captured image from the camera 20, and stores the data in the storage unit 140 sequentially. (Step S104) The inter-frame analysis unit 124 calculates an inter-frame projective transformation matrix that indicates the motion of images between different frames. (Step S106) The coordinate-known point identification unit 126 displays the target image to be processed on the display 30, and identifies image corresponding points for each of the four or more real-space coordinate-known points based on the coordinates indicated by the operation signal input from the operation input unit 160. (Step S108) The real-space inter-image analysis unit 128 determines image corresponding points in the image of the reference frame using the image corresponding points for the real-space coordinate known points identified in any frame and the inter-frame projective transformation matrix. The real-space inter-image analysis unit 128 calculates a real-space projective transformation matrix for the reference frame based on the correspondence between the real-space coordinates of the real-space coordinate known points and the coordinates of the image corresponding points in the reference frame. The real-space inter-image analysis unit 128 calculates a real-space projective transformation matrix for each frame using the inter-frame projective transformation matrix for each frame pair and the real-space projective transformation matrix for the reference frame.

[0035] (Step S110) The calibration unit 130 determines image corresponding points in each frame using an inter-frame projective transformation matrix for each image corresponding point in the reference frame. The calibration unit 130 sets camera parameters related to the real space projective transformation matrix as initial values, and calculates estimated values ​​of image corresponding points estimated using the camera parameters for points with known real space coordinates. The calibration unit 130 determines the difference in coordinate values ​​between the image corresponding points and their estimated values ​​as a reprojection error, and searches for some or all of the camera parameters so as to minimize the reprojection error (minimize the reprojection error). (Step S112) Output processing unit 132 executes processing using the obtained camera parameters. For example, output processing unit 132 corrects the image of that frame using the obtained camera parameters. Output processing unit 132 outputs display data indicating a display screen including the corrected image to display 30, and displays the display screen. Output processing unit 132 transmits the obtained camera parameters to another device, for example, via input / output unit 150. Thereafter, the processing of FIG. 18 ends.

[0036] Next, a specific example of a method for calculating an inter-frame projective transformation matrix according to this embodiment will be described. Method 1: The inter-frame analysis unit 124 may ignore the optical flow calculated for a region where a moving object appears in one or both of the image of the starting frame (hereinafter sometimes referred to as the "starting frame", e.g., the current frame) and the image of the ending frame (hereinafter sometimes referred to as the "ending frame", e.g., the next frame), and may calculate the inter-frame projective transformation matrix using the optical flow of other regions. The inter-frame analysis unit 124 may calculate the inter-frame projective transformation matrix by applying a robust regression method to the optical flow of other regions.

[0037] In the example of Figure 5, optical flows originating from the area in the current frame where a running athlete appears and the area within a certain width from the outer edge are ignored. The inter-frame analysis unit 124 can estimate the area in the target image where a moving object appears. The known image processing technology may be an image recognition technology that recognizes an image area of ​​a moving object such as a human, or the following method may be used. For example, the inter-frame analysis unit 124 calculates the distribution of the number of optical flows for each pair of movement amount and movement direction for the image of the start frame, and determines the pair of movement amount and movement direction with the largest number as the mode of the optical flow. The inter-frame analysis unit 124 considers optical flows whose deviation from the mode is within a certain value to represent the overall movement of the entire image from the start frame to the end frame, and estimates the area associated with the optical flow whose deviation from the mode exceeds a certain value as the area where a moving object appears. For example, when the degree of coincidence between a pattern appearing in the peripheral area of ​​the optical flow start point in the start frame and a pattern appearing in the peripheral area of ​​the optical flow end point in the end frame does not meet a predetermined reference value, the inter-frame analysis unit 124 estimates the area related to the optical flow as an area where a moving object appears. These methods can reduce the amount of processing compared to image recognition technology.

[0038] Method 2: When calculating an inter-frame projective transformation matrix between a source frame and a temporally adjacent destination frame (for example, between adjacent frames), the inter-frame analysis unit 124 may calculate the product of the inter-frame projective transformation matrix from the source frame to the reference frame and the inter-frame transformation matrix from the reference frame to the destination frame as the inter-frame projective transformation matrix from the source frame to the destination frame. The reference frame is a frame that is temporally farther away from the source frame and the destination frame than the interval between them. In the example of FIG. 6, the inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H from the current frame i to the reference frame i+a. i→i+a f and the inter-frame projective transformation matrix H from the reference frame i+a to the next frame i+1. i+a→i+1 f Product H i+a→i+1 f H i→i+a f is the inter-frame projective transformation matrix H i→i+1 f The calculation is performed as follows. Because the deformation and movement of a moving object is relatively small between temporally adjacent frames, it may be difficult to estimate the area in which the object appears. If the optical flow associated with the moving object cannot be excluded when estimating the inter-frame projective transformation matrix, an error will occur in the calculated inter-frame projective transformation matrix. By deriving the inter-frame projective transformation matrix between temporally adjacent frames from the inter-frame projective transformation matrix between frames with a longer time interval, as in this method, the influence of the error can be reduced. Note that the reference frame a may be subsequent to the end frame or prior to the start frame. The reference frame a may be, for example, the start frame or the last frame of a group of frames (described below).

[0039] Method 3: The inter-frame analysis unit 124 may determine the inter-frame projective transformation matrix by restricting the degree of freedom of changes in camera parameters between frames. The inter-frame analysis unit 124 may also adjust the degree of freedom depending on the operation status of the camera 20. The inter-frame projective transformation matrix is ​​a 3x3 matrix with 8 degrees of freedom that is uniquely determined based on the camera parameters. In other words, the degrees of freedom of the inter-frame projective transformation matrix correspond to the degrees of freedom of the camera parameters. Restricting the degrees of freedom of the camera parameters means assuming that only some camera parameters will change, and ignoring changes in other element parameters. The camera parameters are composed of position (3 degrees of freedom), orientation (3 degrees of freedom), and focal length (1 degree of freedom).

[0040] For example, if the movement distance of the camera 20 between frames is less than a predetermined reference value, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix based on a four-degree-of-freedom model in which the orientation and focal length are variable, assuming that the position is constant. The inter-frame analysis unit 124 can estimate the distance traveled by, for example, acquiring signals indicating the camera's motion state detected by position, speed, and acceleration sensors provided in the camera 20, and processing the acquired signals together with the time and time interval of each frame.

[0041] For example, if the range of real space captured by camera 20 is sufficiently far from the position of camera 20 and it is determined that the change in position of camera 20 is relatively small compared to the range of real space captured in the image, inter-frame analysis unit 124 can determine an inter-frame projective transformation matrix based on a model with four degrees of freedom in which the orientation and focal length are variable, under the assumption that the position is constant.

[0042] When no zoom operation is performed, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix under the assumption that the focal length is constant, with only the other parameters being variable. If the position of the camera 20 can be assumed to be constant and no zoom operation is performed, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix based on a three-degree-of-freedom model in which the camera position and focal length are fixed and only the orientation is variable. The inter-frame analysis unit 124 can determine whether or not a zoom operation is being performed, for example, depending on whether or not an operation signal indicating a zoom command to the camera 20 is input. By restricting the degrees of freedom of the camera parameters as described above, the risk of calculating an unrealistic inter-frame projective transformation matrix from an optical flow that contains some error is reduced, and the inter-frame projective transformation matrix is ​​calculated stably with a certain degree of accuracy. Note that the restrictions on the degrees of freedom of the camera parameters will be described later in Appendix 1.

[0043] The inter-frame projective transformation matrices between adjacent frames calculated individually as described above can also be used to calculate an inter-frame projective transformation matrix between temporally separated frames by multiplying them together. However, the inter-frame projective transformation matrix between temporally separated frames calculated in this manner accumulates errors contained in the inter-frame projective transformation matrices between the individual adjacent frames. The inter-frame analysis unit 124 can reduce the accumulated errors by applying any one or a combination of the processes related to the following methods 4 to 7.

[0044] Method 4: The inter-frame analysis unit 124 may search for an inter-frame projective transformation matrix for each pair of frames in a group of frames, such that the inter-frame projective transformation matrix obtained by multiplying the optical flow between any two frames (e.g., the first frame to the last frame of the group of frames) by the inter-frame projective transformation matrix between other frames has a high degree of concordance with the inter-frame projective transformation matrix between the frames. In the example of Figure 7, the inter-frame projective transformation matrix from the first frame i to the last frame i+l of the group of frames is calculated as the total product of the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames, and the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames is calculated as the estimated inter-frame projective transformation matrix from the first frame i to the last frame i+l, and the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames is calculated as the estimated inter-frame projective transformation matrix from the first frame i to the last frame i+l, with the aim of increasing the degree of concordance between the optical flow between the two frames.

[0045] The frame section to be evaluated for compatibility is not limited to a section beginning with the start frame and ending with the last frame. This section may be any section bounded by two different frames within a frame group. FIG. 8 illustrates an example in which the product of the sum of the inverse matrices of the inter-frame projective transformation matrices for each pair of adjacent frames from the start frame i to the penultimate frame i+l-1 and the inter-frame projective transformation matrix from the start frame i to the final frame l is used as the estimated value for the inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l. The inter-frame analysis unit 124 searches for the inter-frame projective transformation matrix for each pair of adjacent frames from the start frame i to the penultimate frame i+l-1 in the frame group and the inter-frame projective transformation matrix from the start frame i to the final frame l, while taking into account the need to increase the compatibility between the estimated inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l and the optical flow between the two frames.

[0046] Assuming that the inter-frame projective transformation matrices for some of the frame pairs are default, the inter-frame analysis unit 124 may perform the above-mentioned inter-frame projective transformation matrix search for only the remaining frame pairs, and may leave the inter-frame projective transformation matrices for some of the frame pairs fixed to the default values ​​and exclude them from the inter-frame projective transformation matrix search. 9, like the example of Fig. 7, is based on the process of calculating the sum of the inter-frame projective transformation matrices for each pair of adjacent frames in the frame group as an estimate of the inter-frame projective transformation matrix from start frame i to final frame i+l, and then searching for an inter-frame projective transformation matrix for each pair of adjacent frames that matches the optical flow between the two frames more closely. Here, the inter-frame analysis unit 124 only searches for the inter-frame projective transformation matrix from frame i2 to frame i2+1, and regards the inter-frame projective transformation matrices for each pair of other frames as default values.

[0047] 8, the example of Fig. 10 is based on the inter-frame analysis unit 124 searching for an inter-frame projective transformation matrix for each pair of adjacent frames from the start frame i to the next-to-last frame i+l-1 in the group of frames, and an inter-frame projective transformation matrix from the start frame i to the final frame l, so as to achieve a higher degree of compatibility between the estimated inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l and the optical flow between the two frames. Here, the inter-frame analysis unit 124 searches only for the inter-frame projective transformation matrix from frame i2 to frame i2+1, and regards the inter-frame projective transformation matrices for each other pair of frames as default values. As the inter-frame projective transformation matrix assumed to be the default, for example, a substantially accurate inter-frame projective transformation matrix, that is, an inter-frame projective transformation matrix obtained in advance with a degree of precision that is expected to be accurate, is applied.

[0048] In Method 4, the longer the period from the start frame i to the final frame i+l, the greater the effect of reducing cumulative error. However, to calculate an optical flow that can accurately estimate the inter-frame projective transformation matrix between both frames, the ratio (hereinafter referred to as the "image overlap rate") of the area in the image where a common area in real space appears between both frames (hereinafter referred to as the "common area") must be at least a certain level. Therefore, the inter-frame analysis unit 124 identifies the common area after the fact based on the inter-frame projective transformation matrix searched for each pair of different frames. The inter-frame analysis unit 124 may define a series of frames as a single interval in which the identified common area occupies a certain ratio or more. The image overlap rate of the common area is set in advance based on a balance with the influence of errors contained in the optical flow. A series of frames consisting of multiple frames in which a common scene appears is determined based on the image overlap rate of the common area. The inter-frame analysis unit 124 may execute Method 4 for each predetermined frame group, or may perform the search for the inter-frame projective transformation matrix and the setting of the frame group in parallel. Note that common frames may exist among multiple frame groups. In this case, overlapping of time series between the frame groups is permitted. By executing Method 4 for each frame group, the inter-frame analysis unit 124 can reduce accumulated errors across multiple frame groups.

[0049] Method 5: The inter-frame analysis unit 124 may perform a small transformation on the inter-frame projective transformation matrices of some or all of the frame pairs in the frame group, and then check the compatibility of the transformed inter-frame projective transformation matrices with the optical flow according to Method 4 to search for an inter-frame projective transformation matrix that has higher compatibility. The projective transformation matrix is ​​expressed as the product of element matrices indicating the components of each element parameter (see Supplementary Note 1). Element parameters refer to individual parameters that are elements of the camera parameters. Examples of element parameters include the position, orientation, and focal length of the camera 20. In the infinitesimal transformation, the inter-frame analysis unit 124 assumes that some or all of the element parameters have fluctuated randomly and infinitesimally, and can calculate the product of element matrices corresponding to the amount of fluctuation related to that fluctuation as the infinitesimal transformation matrix ΔH.

[0050] 11 illustrates a case in which, in the method illustrated in FIG. 7, each interframe projective transformation matrix between adjacent frames is multiplied by an infinitesimal transformation matrix to search for an interframe projective transformation matrix that improves the sum of the compatibility between the interframe projective transformation matrix and the optical flow between adjacent frames and the compatibility between the interframe projective transformation matrix and the optical flow between the start frame and the final frame. Here, the interframe analysis unit 124 compares compatibility 1 obtained using the interframe projective transformation matrix before transformation with compatibility 2 obtained using the interframe projective transformation matrix after transformation. If compatibility 2 is higher than compatibility 1, the interframe projective transformation matrix is ​​updated to the post-transformation interframe projective transformation matrix as the new interframe projective transformation matrix. Otherwise, the interframe projective transformation matrix is ​​not updated. The interframe analysis unit 124 then repeats the infinitesimal transformation of the interframe projective transformation matrix, comparing the compatibility before and after transformation, and updating the interframe projective transformation matrix until the compatibility no longer improves or a predetermined number of repetitions is reached. When roughly accurate interframe projective transformation matrices are obtained between all adjacent frames in a frame group, it may be difficult to further reduce the accumulated error that appears in the interframe projective transformation matrices between temporally distant frames, even by repeatedly using robust regression methods such as the RANSAC algorithm. Even in such cases, adding random small transformations can further search for interframe projective transformation matrices that would not be identified by the above methods, thereby providing an opportunity to reduce the accumulated error.

[0051] If an ideal value (or target value) of the inter-frame projective transformation matrix between two frames in a certain frame group is set or known, the inter-frame analysis unit 124 may make small variations in the inter-frame projective transformation matrix between other frames based on the ideal value to check whether the degree of conformance with the optical flow improves. Here, the inter-frame analysis unit 124 calculates the difference H between the ideal value and the current value. diff and distributes it to each pair of frames to calculate the difference component dH. The inter-frame analysis unit 124 corrects the inter-frame projective transformation matrix by applying the difference component dH distributed to the pair of frames (see FIG. 12). The inter-frame analysis unit 124 then determines whether the corrected inter-frame projective transformation matrix improves the compatibility with the optical flow related to the group of frames compared to the compatibility based on the inter-frame projective transformation matrix before correction. If the inter-frame analysis unit 124 determines that the compatibility will improve, it updates the inter-frame projective transformation matrix to the corrected one, and if it determines that the compatibility will not improve, it does not update the inter-frame projective transformation matrix to the corrected one. Note that minute transformation will be described later in Supplementary Note 2.

[0052] Method 6: For each pair of adjacent frames to be processed, the inter-frame analysis unit 124 may calculate a weighted average of inter-frame projective transformation matrices between multiple frames that are consecutive in time series, with the target frames as the reference. The inter-frame analysis unit 124 calculates the compatibility between the inter-frame projective transformation matrix and the optical flow before and after calculating the weighted average. If the sum of the calculated compatibility between a pair of frames does not decrease due to the weighted average, the inter-frame analysis unit 124 updates the weighted average as a new inter-frame projective transformation matrix. If the sum of the calculated compatibility decreases due to the weighted average, the inter-frame analysis unit 124 adopts the inter-frame projective transformation matrix before calculating the weighted average. This allows the inter-frame analysis unit 124 to accurately estimate image motion while reducing fluctuations in the projective transformation matrix between frames. The weighted average will be described later in Supplementary Note 3.

[0053] Method 7: The inter-frame analysis unit 124 may determine whether a common region representing a common real space is shared even between frames that are not consecutive in time, and may execute any of methods 4 to 6 between frames that share a common region. Fig. 13 shows a case where a group of frames (shown as three rectangular frames on the left and one rectangular frame on the right in Fig. 13) that share a common region are generated at intervals in time, and the inter-frame projective transformation matrices of an intermittent group of frames that intermittently connect these frames are calculated in chronological order as H0 f ,… , H4 f The inter-frame analysis unit 124 calculates the total product H intermit f may be replaced with the total product of the inter-frame projective transformation matrices between all adjacent frames during this period, and any one of methods 4 to 6 may be performed. After performing a process to calculate or update the inter-frame projective transformation matrices between frames included in the intermittent frame group, the inter-frame analysis unit 124 may set the inter-frame projective transformation matrix obtained by this process as a default value and perform any one of methods 4 to 6 on all adjacent frames included between each of the frames (see FIG. 14 ).

[0054] Note that, before performing processing on the intermittent frame groups as illustrated in Figures 13 and 14, the inter-frame analysis unit 124 may perform any of methods 4 to 6 on each consecutive frame group that includes those frames. When determining a frame as one that shares a common area, the inter-frame analysis unit 124 can determine, for example, that a frame included in a certain frame group has a common area shared with another frame where the ratio of the portion that shares the common area with that frame group is equal to or greater than a predetermined ratio, as a frame that shares a common area with that frame group. When capturing an image, a user may point the camera 20 from one real space region to another, and then re-point it back to the original real space region. In such a case, multiple groups of temporally separated frames representing the original real space region are generated, and cumulative error occurs in the inter-frame projective transformation matrix connecting each frame. This method can reduce cumulative error between non-consecutive frames representing a common real space region.

[0055] The inter-frame analysis unit 124 may execute any one of the processes of methods 1 to 7, or may execute a combination of some or all of them as long as no contradiction occurs. The inter-frame analysis unit 124 may execute the process of determining a frame group and the process of calculating an inter-frame projective transformation matrix independently or sequentially, or may execute both processes synchronously. Next, an example of the calculation process of the inter-frame projective transformation matrix will be described. Fig. 19 is a flowchart illustrating the calculation process of the inter-frame projective transformation matrix. Fig. 19 illustrates a case where the step of determining a frame group and the step of calculating the inter-frame projective transformation matrix are synchronized.

[0056] (Step S202) The inter-frame analysis unit 124 initially sets, as a group of consecutive frames, a predetermined number of frames that are adjacent in time and are fully expected to have an image overlap rate in the common area equal to or greater than a predetermined rate from the images of each frame shown in the image data. The inter-frame analysis unit 124 provisionally calculates an inter-frame projective transformation matrix between adjacent frames using Method 2 for each frame in the group of consecutive frames. The inter-frame analysis unit 124 then calculates the optical flow from the image of the first frame to the image of the last frame of the group of consecutive frames, and calculates the inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the calculated inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the degree of conformance 1 between the inter-frame projective transformation matrices of adjacent frames in the continuous frame group, and calculates the cumulative value H of the inter-frame projective transformation matrices from the start frame to the final frame. f-multi The inter-frame analysis unit 124 calculates the calculated cumulative value H f-multiand calculates the degree of conformance 2 between the optical flow. The inter-frame analysis unit 124 compares degree of conformance 1 and degree of conformance 2, and if degree of conformance 2 is lower than degree of conformance 1, performs the processing of method 5 on the group of consecutive frames to adjust the inter-frame projective transformation matrix between adjacent frames. Thereafter, the inter-frame analysis unit 124 performs the processing of method 6 to further smooth the inter-frame projective transformation matrix between adjacent frames across multiple frames.

[0057] (Step S204) The inter-frame analysis unit 124 determines the frame immediately following the group of consecutive frames as the next frame, and provisionally calculates an inter-frame projective transformation matrix from the last frame of the group of consecutive frames according to Method 2. (Step S206) The inter-frame analysis unit 124 calculates the optical flow from the image of the starting frame of the consecutive frame group to the image of the next frame, and calculates the inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the calculated inter-frame projective transformation matrix H f-long The degree of conformance between the optical flow and the object is calculated.

[0058] (Step S208) The inter-frame analysis unit 124 calculates the total product of the inter-frame projective transformation matrix between adjacent frames in the group of consecutive frames and the inter-frame projective transformation matrix from the last frame of the group of consecutive frames to the next frame, to obtain a cumulative value H f-multi The inter-frame analysis unit 124 calculates the calculated cumulative value H f-multi and calculates the goodness of fit 2 between the optical flow. The inter-frame analysis unit 124 compares goodness of fit 1 and goodness of fit 2, and if goodness of fit 2 is lower than goodness of fit 1, it performs the process of method 5 on the final frame of the consecutive frame group and the next frame to adjust their inter-frame projective transformation matrices. At this stage, it is not necessary to consider the movements from the first frame of the consecutive frame group to the frame immediately before the final frame. The inter-frame analysis unit 124 then recalculates goodness of fit 2.

[0059] (Step S210) The inter-frame analysis unit 124 defines an area in which the start frame and the next frame in the group of consecutive frames show a common image as a common area, and calculates the ratio of the defined common area to the entire image as the image overlap rate. The inter-frame analysis unit 124 determines whether the calculated image overlap rate is equal to or lower than a predetermined level. If the image overlap rate is equal to or lower than the predetermined level (YES in step S210), the inter-frame analysis unit 124 proceeds to processing in step S216. In this case, the inter-frame analysis unit 124 determines that the next frame cannot be included in the group of consecutive frames immediately before it. This confirms one group of consecutive frames. If the image overlap rate exceeds the predetermined level (NO in step S210), the inter-frame analysis unit 124 proceeds to processing in step S212.

[0060] (Step S212) The inter-frame analysis unit 124 determines whether or not the matching score 2 is equal to or lower than a predetermined level. If the matching score 2 is equal to or lower than the predetermined level (YES in step S212), the process proceeds to step S216. In this case, the inter-frame analysis unit 124 also determines that the next frame cannot be included in the immediately preceding group of consecutive frames. If the matching score 2 is higher than the predetermined level (NO in step S212), the process proceeds to step S214. (Step S214) The inter-frame analysis unit 124 adds the next frame as the final frame to the group of consecutive frames, and then proceeds to the processing of step S204.

[0061] (Step S216) The inter-frame analysis unit 124 searches for an angle-of-view overlap frame whose angle of view overlaps with the angle of view of the start frame of the determined consecutive frame group from past frames that precede the consecutive frame group. An angle-of-view overlap frame is a frame related to an image that represents a common field of view, that is, a frame for which the area ratio of the common area with the start frame is equal to or greater than a certain level and for which an inter-frame projective transformation matrix is ​​obtained whose matching rate with the optical flow from the start frame is equal to or greater than a certain level. If an angle-of-view overlap frame is found (step S216 YES), the process proceeds to step S218. If an angle-of-view overlap frame is not found (step S216 NO), the process of FIG. 19 ends. (Step S218) The inter-frame analysis unit 124 identifies frames with overlapping angles of view and performs the process of method 7 on the intermittent frame group including the frames in the latest consecutive frame group to reduce (minimize) the error in the inter-frame projection matrices belonging to the intermittent frame group. Furthermore, for consecutive adjacent frames existing between each frame included in the intermittent frame group, the process of methods 4 to 6 is used to adjust these inter-frame projection transformation matrices so that they conform to the inter-frame projection transformation matrices between each frame included in the intermittent frame group. Then, the process of FIG. 19 ends.

[0062] Appendix 1. Use of a projective transformation matrix with limited degrees of freedom Appendix 1-1. About the Projection Transformation Matrix The transformation of a point between the coordinates representing its position in one coordinate system and the coordinates representing its position in another coordinate system, or the movement from one point to another in one coordinate system, or the correspondence between two points, is mathematically expressed as a projective transformation. The matrix that represents this projective transformation is called a projective transformation matrix. For example, the relationship between a point m on a two-dimensional coordinate system and a point m′ after projective transformation is expressed using a projective transformation matrix H and equation (1).

[0063]

number

[0064] In equation (1), the coordinates of points m and m' are expressed in the homogeneous coordinate system by equations (2) and (3). The projective transformation matrix H is a 3x3 matrix expressed by equation (4). λ is a scale coefficient. λ can take any scalar value. Because of this arbitrariness, any element, for example, h 22 The projective transformation matrix H may be normalized by setting the value of ∑ to a constant value other than 0 (for example, 1). Therefore, the degree of freedom of the projective transformation matrix H is one less than the number of elements. The degree of freedom of the projective transformation matrix H exemplified by equations (1) and (4) is 8.

[0065]

number

[0066]

number

[0067]

number

[0068] Appendix 1-2. Restrictions on the degrees of freedom of the projective transformation matrix In camera geometry, the transformation from a point M on a three-dimensional real space coordinate system to a point m on a camera image coordinate system is expressed using a projective transformation matrix P as shown in equation (5).

[0069]

number

[0070] In equation (5), the coordinates of points M and m are expressed by equations (6) and (7), respectively. The projective transformation matrix P is a 3-row, 4-column matrix expressed by equation (8).

[0071]

number

[0072]

number

[0073] The projective transformation matrix P is expressed as the product of element matrices A, F, R, and [E -T], which indicate the contribution of element parameters, as shown in equation (8). The element matrix A is determined by the skew distortion s, the aspect ratio ρ, and the coordinates (u0, v0) of the intersection between the virtual image plane of the camera image and the optical axis of the camera, as shown in equation (9). Depending on the specifications of the camera 20, s = 0, ρ = 1, (u0, v0) may be approximated as the coordinates of the center of the image. Alternatively, the values ​​of s, ρ, u0, and v0 may be determined in advance using a known camera internal parameter estimation technique. The element matrix F is determined by the focal length f as shown in equation (10). The element matrix R is a rotation matrix with three degrees of freedom. The element matrix R is determined by the orientation of the camera in the real space coordinate system as shown in equation (11). The vector T is determined by the coordinates (t x , t y , y z ) The matrix E is a 3-row, 3-column identity matrix.

[0074]

number

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079] Regarding the positions m0 and m1 on the camera image coordinate system projected in two different frames of point M on the real-time coordinate system, the relationship shown in equation (13) is derived from equations (5) and (8). In equation (13), 0 and 1 indicate either of the two frames. The element matrices F and R and the vector T are variable depending on the frame.

[0080]

number

[0081] If the real space coordinate system is set to match the position and orientation of the camera in frame 0, and the scale of the real space coordinate system is set so that the focal length f0 in frame 0 is f0=1, then T0=[0,0,0] T , R0 = E, F0 = E. Furthermore, if we assume that the camera position is constant in the real space coordinate system during shooting, i.e., T0 = T1, then equation (13) can be rearranged as equation (14). Therefore, the projective transformation matrix H from frame 0 to frame 1 is given by equation (15). Since the element matrix A is an internal parameter that can be acquired from the camera 20 by the image processing device 10 or set for the camera 20, the degrees of freedom of the projective transformation matrix H are four in total: one degree of freedom for the element matrix F1 and three degrees of freedom for the rotation matrix R1. Furthermore, if no zoom operation is performed during shooting, the element matrix F1 becomes equal to the element matrix F0 (which can be considered as F1 = E), and the degrees of freedom of the projective transformation matrix H are further reduced to three. In this projective transformation matrix H with limited degrees of freedom, the components due to unrestricted, variable parameters change, while the components due to restricted, fixed parameters do not change.

[0082]

number

[0083]

number

[0084] Appendix 1-3. Quantification of displacement between camera images using a projective transformation matrix with restricted degrees of freedom The inter-frame projective transformation used in this application can sometimes be expressed with fewer degrees of freedom than the original degrees of freedom, depending on the image capture conditions. Using a projective transformation matrix with limited degrees of freedom can prevent unrealistic image transitions from being detected, allowing for stable estimation of the projective transformation matrix. When changes in camera position during capture can be ignored and changes in camera orientation and zoom are allowed, the inter-frame analysis unit 124 can use a projective transformation matrix with four degrees of freedom, modeled as in Equation (16).

[0085]

number

[0086] If changes in zoom as well as camera position can be ignored during shooting and changes in camera orientation are allowed, the inter-frame analysis unit 124 can use a projective transformation matrix with three degrees of freedom modeled as in equation (17).

[0087]

number

[0088] When determining the inter-frame homography matrix, the inter-frame analysis unit 124 calculates the image transition between target frames as an optical flow for each block. The inter-frame analysis unit 124 can then estimate the inter-frame homography matrix based on the above model using a regression method such as the RANSAC algorithm so as to increase the degree of conformance with the calculated optical flow. Specifically, the inter-frame analysis unit 124 can calculate the inter-frame homography matrix by executing the procedure illustrated in FIG. 20.

[0089] The inter-frame analysis unit 124 repeats the processing of steps S302 to S306 a predetermined number of times, and then proceeds to the processing of step S308. (Step S302) The inter-frame analysis unit 124 randomly selects optical flows in a number corresponding to the degrees of freedom of the model. For example, when using models with four degrees of freedom and three degrees of freedom, the number of optical flows is two. (Step S304) The inter-frame analysis unit 124 applies a model to the selected optical flow. That is, the inter-frame analysis unit 124 determines parameters of the model so that the movement indicated by the selected optical flow is as close as possible to the movement given by the inter-frame projective transformation matrix calculated using the model.

[0090] (Step S306) The inter-frame analysis unit 124 determines whether the motion indicated by the other optical flows not selected matches the motion given by the inter-frame projective transformation matrix calculated using a model under the determined parameters. The inter-frame analysis unit 124 can determine whether the motion matches based on, for example, whether the difference between the two types of motion is equal to or less than a predetermined reference value. The inter-frame analysis unit 124 counts the number of other optical flows whose motion matches as the degree of match. (Step S308) The inter-frame analysis unit 124 determines the inter-frame projective transformation matrix that gives the highest degree of conformance as the regression value, and then ends the processing of FIG.

[0091] In addition, when the position of the camera 20 is not changed but the orientation and zoom are changed, and the subject is sufficiently far away from the camera 20, it can be assumed that the image transition between frames is composed of elements of translation (2 degrees of freedom, parameters: Δu, Δv), rotation (1 degree of freedom, parameter α), and enlargement or reduction (1 degree of freedom, parameter f). Therefore, the homography matrix can be approximated as shown in equation (18). In this case, the number of optical flows to be randomly extracted in the regression is two. In addition, if it is assumed that no enlargement or reduction occurs, the parameter f is fixed to 1.

[0092]

number

[0093] Instead of parameter optimization, the derivation of parameters that fit the two optical flows is analytically realized using a closed-form solution, which will be explained next, thereby reducing the computational cost. Coordinates m0([u0, v0] on the camera image system T ) to coordinate m1([u1, v1] T ) is rearranged as shown in equations (19) to (22) using the projective transformation matrix H shown in equation (18).

[0094]

number

[0095]

number

[0096]

number

[0097]

number

[0098] The coordinates m0 and m1 before and after the transformation are obtained from each of the two randomly extracted optical flows. 00 , m 10 and the coordinates m before and after the transformation for the other optical flow 1. 01 , m 11 are expressed by equations (23) and (24). By substituting equations (23) and (24) into equation (22) and solving them simultaneously, equation (25) is obtained, which is further transformed into equation (26). According to equation (26), the inter-frame analysis unit 124 can determine the scaling factor f, the rotation angle α, and the translation amount (Δu, Δv) from the two optical flows.

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103] On the other hand, if the position of camera 20 changes during shooting, the degrees of freedom required to express that movement increase by three, resulting in seven degrees of freedom, including one for the focal length and three for the camera orientation. These degrees of freedom are close to the eight degrees of freedom of the 3-by-3 projective transformation matrix H shown in equation (27), and can be directly applied to this projective transformation matrix H. For the application, it is necessary to extract optical flows at four points. Even in this case, the inter-frame analysis unit 124 can analytically calculate the parameters using a closed-form solution.

[0104]

number

[0105] Appendix 2. Use of a projective transformation matrix to represent small movements The projective transformation matrix representing the minute movement refers to the above-mentioned infinitesimal transformation matrix ΔH. The infinitesimal transformation matrix ΔH is derived by randomly varying at least some element parameters belonging to the camera parameters. In this application, we propose adding the infinitesimal transformation matrix ΔH to the inter-frame projective transformation matrix H and checking the compatibility of the inter-frame projective transformation matrix H after adding the infinitesimal transformation matrix ΔH with the optical flow to search for an inter-frame projective transformation matrix H with higher compatibility with the optical flow. In a search method using a regression technique such as RANSAC from the optical flow, the true inter-frame projective transformation matrix cannot be guaranteed unless data with strictly true values ​​can be sampled. However, by performing the following method to expand the range of inter-frame projective transformation matrix candidates searched, compatibility with the optical flow and, ultimately, the accuracy of the inter-frame projective transformation matrix H can be improved.

[0106] The inter-frame analysis unit 124 generates the infinitesimal transformation matrix ΔH using a model that uses camera parameters as explanatory variables. The inter-frame analysis unit 124 selects a model with degrees of freedom corresponding to the shooting conditions. The inter-frame analysis unit 124 can acquire data such as operation information such as zooming from the camera 20, and sensor signals indicating the movement state or orientation, and determine the shooting conditions based on the acquired data. For example, when the position of the camera 20 does not change but the zoom changes, the inter-frame analysis unit 124 generates the infinitesimal transformation matrix ΔH using equation (28). Equation (28) corresponds to a model similar to equation (16) and includes an element matrix F1 (one degree of freedom) and a rotation matrix R1 (three degrees of freedom) as variable factors. Therefore, the infinitesimal transformation matrix ΔH has four degrees of freedom. Note that when the inter-frame analysis unit 124 determines that no zoom change will occur, it fixes the element matrix F1, which includes the focal length, to F1=E, and allows only the rotation matrix R1 to be variable. In equation (28), when F1=E and R1=E, the infinitesimal transformation matrix ΔH becomes a unit matrix, which indicates that no movement of the image occurs.

[0107] The inter-frame analysis unit 124 generates minute random noise centered on a value that gives element matrices F1 = E, R1 = E for the variable element parameters, and generates a minute transformation matrix ΔH containing the element matrices F1 and R1 as factors based on the generated random noise. For example, the inter-frame analysis unit 124 can generate the minute random noise by normalizing each sample value constituting a pseudo-random number sequence generated using a known method so that the sample values ​​fall within a predetermined center value and range. The range is set to be larger than calculation errors (e.g., quantization error, pixel spacing) but sufficiently smaller than typical fluctuations during shooting. If the inter-frame analysis unit 124 determines that no zoom change will occur, it fixes F1 = E, i.e., fixes the focal length f to 1, and allows only the orientation to be variable.

[0108]

number

[0109] The inter-frame analysis unit 124 may generate the infinitesimal transformation matrix ΔH using the model shown in equation (29). The model shown in equation (29) is the same as the model in equation (18) and has four degrees of freedom. In equation (29), if Δu=0, Δv=0, α=0, and f=1, the infinitesimal transformation matrix ΔH becomes a unit matrix, indicating that no image movement occurs. Then, infinitesimal random noise with central values ​​of 0, 0, 0, and 1 is generated for each element parameter Δu, Δv, α, and f, respectively, and the infinitesimal transformation matrix ΔH is generated using equation (29) based on the generated random noise. Note that when the inter-frame analysis unit 124 determines that no zoom change will occur, it fixes the focal length f to 1 and makes only Δu, Δv, and α variable.

[0110]

number

[0111] In addition, when it is already known how to improve the inter-frame projective transformation matrix H, that is, when its ideal value or target value is set, the inter-frame analysis unit 124 calculates the amount of change H required for this improvement before performing the minute transformation on the inter-frame projective transformation matrix H. diff If a pair of two frames related to the inter-frame projective transformation matrix H are adjacent, the inter-frame analysis unit 124 may apply a change amount H to the inter-frame projective transformation matrix H that has already been obtained. diff and the product is determined as the corrected inter-frame projective transformation matrix H. When the pair of two frames related to the inter-frame projective transformation matrix H is separated in time by two or more frames, the inter-frame analysis unit 124 calculates the amount of change H diff is distributed to the difference component dH for each pair of adjacent frames, and multiplied by the difference component dH for each pair of adjacent frames to correct the inter-frame projective transformation matrix H for each pair of adjacent frames. For example, the ideal inter-frame projective transformation matrix H from the start frame i to the final frame i+l in a frame group is i→i+l f is known, the inter-frame analysis unit 124 calculates the current value H of the inter-frame projective transformation matrix at that time as shown in equation (30). i→i+l f ' is multiplied by the ideal value to obtain the change H diff Calculate.

[0112]

number

[0113] Then, the inter-frame analysis unit 124 calculates the calculated variation H diff is calculated as the difference component dH to be distributed to the inter-frame projective transformation matrix of each set of l-1 adjacent frames in the frame group. Here, when the inter-frame projective transformation matrix is ​​derived using a reference frame, the set of adjacent frames from the starting frame to the reference frame is also calculated as the difference component H diffFor example, the inter-frame analysis unit 124 can calculate the difference component dH using the model of equation (31). The model of equation (31) is the same as equation (28), and has four degrees of freedom.

[0114]

number

[0115] The inter-frame analysis unit 124 calculates the difference H diff The inter-frame analysis unit 124 divides the vectors m0 and m1 shown in equation (32) into the difference component dH for each pair of adjacent frames as follows, for example. diff Multiplying this by m i The inter-frame analysis unit 124 calculates the total product Π of the difference component dH for the number of pairs of adjacent frames in the frame group as shown in equation (34). l Multiplying by dH gives the estimated movement amount m i The inter-frame analysis unit 124 determines the target movement amount m i ' and estimated movement amount m i A difference component dH for each pair of adjacent frames that minimizes the sum of squared differences of " is searched for. Note that f1 in equation (35) represents a focal length parameter that constitutes the element matrix F1.

[0116]

number

[0117]

number

[0118]

number

[0119]

number

[0120] Appendix 3. Weighted moving average (smoothing) of the homography matrix over consecutive frames The weighted moving average is equivalent to calculating a weighted sum of element values ​​for each element of the inter-frame projective transformation matrix, assuming similarity between consecutive adjacent frames. In the example of equation (36), the weight for the target frame is generally (1-2α), and the weights for the previous frame immediately before it and the next frame immediately after it are each set to α. The coefficient α indicates the degree of smoothing effect achieved by the moving average. The coefficient α is a real number greater than 0 and less than 0.5. However, since the previous frame is not taken into account for start frame 0 in the frame group, the weight for start frame 0 is set to 1-α. Since the subsequent frame is not taken into account for final frame l, the weight for final frame l is set to 1-α.

[0121]

number

[0122] The inter-frame projective transformation matrix H smoothed by the weighted moving average obtained based on Eq. (36) i ' does not necessarily represent a weighted average of a geometrically strict projective transformation. Therefore, when performing the moving average, the inter-frame analysis unit 124 uses the inter-frame projective transformation matrix H obtained by the moving average. i The compatibility between ' and the optical flow between the relevant frames is evaluated, and if the compatibility improves, the inter-frame projective transformation matrix H obtained by moving average is used. i ' is adopted, and if the compatibility does not improve, the original inter-frame projective transformation matrix H i Specifically, the inter-frame analysis unit 124 uses the individual inter-frame projective transformation matrices H i The procedure shown in FIG. 21 is executed for

[0123] (Step S402) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i The moving average value H is calculated based on a series of inter-frame projective transformation matrices within a predetermined interval including i ' is estimated. (Step S404) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i Moving average value H i The degree of conformance 1 between ' and the optical flow between the frames is evaluated. (Step S406) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i The degree of conformance between the optical flow and the frame is evaluated.

[0124] (Step S408) The inter-frame analysis unit 124 compares the fitness 1 with the fitness 2, and determines whether the fitness 1 is lower than the fitness 2. If the fitness 1 is lower (YES in step S408), the original inter-frame projective transformation matrix H i is adopted, and the processing in Fig. 21 is terminated. If the fitness 1 does not decrease (NO in step S408), the processing proceeds to step S410. (Step S410) The inter-frame analysis unit 124 calculates the moving average value H i ' is used to calculate the inter-frame projective transformation matrix H i Update as. (Step S412) The inter-frame analysis unit 124 determines whether the number of repetitions of steps S404 to S410 has reached a predetermined number of repetitions. If it is determined that the number has not reached (step S412 YES), the process returns to step S404. If it is determined that the number has reached (step S412 NO), the process of FIG. 21 ends.

[0125] The inter-frame analysis unit 124 does not necessarily need to perform the moving average for all inter-frames. The inter-frame analysis unit 124 may perform the moving average for the inter-frame projective transformation matrices for some inter-frames (for example, between any one frame or more and less than l frames in one frame group).

[0126] Appendix 4. Projection transformation matrix between real space plane coordinate system and camera image coordinate system (real space projective transformation matrix) H g Derivation of The coordinate transformation from point M on the real space coordinate system to point m on the camera image coordinate system is expressed by equations (37) to (40) using a projective transformation matrix P with 3 rows and 4 columns.

[0127]

number

[0128]

number

[0129]

number

[0130]

number

[0131] Assuming that the position of point M exists on the plane z=0 in real space, the relationships shown in equations (37) to (40) can be rearranged as shown in equations (41) and (42).

[0132]

number

[0133]

number

[0134] By omitting the third column of the projection transformation matrix P and the z coordinate of point m in equation (42), a 3-by-3 real space projection transformation matrix H g is derived.

[0135]

number

[0136]

number

[0137]

number

[0138] In equation (45), p 23 When normalized as =1, the relationships shown in equations (46) and (47) are derived.

[0139]

number

[0140]

number

[0141] Here, for n points with known coordinates arranged on a real space plane, the image coordinate values ​​(u k , v k ) (k=0, ..., n-1) is obtained, equation (47) can be expressed as a simultaneous equation in equation (48). If the matrix of the first term on the left side of equation (48) is B as in equation (49) and the vector on the right side is C as in equation (50), equation (48) can be rearranged as in equation (51). Equation (51) is 23 = 1, and convert matrix B and vector C into real space projection matrix H g is calculated. T B) has an inverse matrix, its rank must be 8. Therefore, if there are four or more known coordinate points, the real space projection transformation matrix H g It is shown that it is possible to calculate

[0142]

number

[0143]

number

[0144]

number

[0145]

number

[0146] In addition, the real space projection transformation matrix H g The calculation of may be performed by applying a known method. For example, when six or more points with known real space coordinates, including points not on a plane, can be identified in the standard frame, the projection transformation matrix P is calculated from the relationship between the real space coordinate values ​​of these known real space coordinate points and the camera image coordinate values ​​of the corresponding image points. Based on the element values ​​of this P and equation (45), H g Calculate.

[0147] Then, as shown in equation (54) derived from equations (52) to (53), the real space projection transformation matrix H g For the inter-frame projective transformation matrix H from the reference frame to other frames, f By multiplying by , the real space projection transformation matrix H g Therefore, the real space projection transformation matrix H g and for every pair of frames, the inter-frame projective transformation matrix H f is obtained, the real space projection transformation matrix H g is obtained.

[0148]

number

[0149]

number

[0150]

number

[0151] Appendix 5. Derivation of the projective transformation matrix P between the real space coordinate system and the camera image coordinate system Appendix 5-1. Calculation of focal length and external camera parameters Real space projection transformation matrix H g is expressed as a product of element matrices A, F, R, etc. using the camera parameters as shown in equation (55). The relationship shown in equation (56) holds for the first and second columns on both the left and right sides of equation (55). Then, the elements of the first and second columns of the rotation matrix R are expressed as a column vector r1 T , r2 T Then, equation (56) is transformed into equation (57). In this example, the parameters s and ρ constituting element matrix A are approximated as s = 0 and ρ = 1, but these parameters may be set to predetermined values.

[0152]

number

[0153]

number

[0154]

number

[0155] From the properties of the rotation matrix R, the column vector r0 T , r1 T is r0 T r0=r1 T r1, r0 TThe relationship r1 = 0 is satisfied. According to this relationship, equation (58) can be obtained from equation (57). By transferring the focal length f to the left side of equation (58), two equations (59) related to f are obtained.

[0156]

number

[0157]

number

[0158] The calibration unit 130 may calculate the focal length f using either of the two equations in equation (59), but the focal length f may differ for each equation due to the influence of errors related to the processing up to this point. Therefore, the calibration unit 130 may calculate the average value of both as the focal length f. For example, the calibration unit 130 may calculate f obtained from each of the two equations as shown in equation (60). 2 The square root of the average value of the above may be calculated as the focal length f.

[0159]

number

[0160] The calibration unit 130 can calculate the rotation matrix R and the element matrix T according to equations (61) to (67) using the obtained focal length f. The rotation matrix R indicates the orientation of the camera 20. The element matrix T indicates the position of the camera 20. The element matrix F is a matrix configured from the focal length f of the camera 20.

[0161]

number

[0162]

number

[0163]

number

[0164]

number

[0165]

number

[0166]

number

[0167]

number

[0168] Appendix 5-2. Provisional calculation of camera position and recalculation of focal length and camera direction The focal length f and camera position calculated using the method described in Appendix 5-1 may contain errors and differ significantly from the true values. On the other hand, the error between the camera orientation and the direction of the camera optical axis vector is relatively small. Furthermore, camera position errors tend to occur in the direction of the optical axis vector, which is related to focal length errors. Therefore, assuming that the camera position does not change during shooting and that the camera position is on the optical axis (see FIG. 17), the calibration unit 130 provisionally calculates candidate camera positions. Here, any coordinate on the optical axis is formulated as being at position T+kr^2, which is k times the displacement indicated by the third row vector r^2 of the rotation matrix R (corresponding to a vector parallel to the camera optical axis) from the camera position T. The distance D between the coordinate T0 of the candidate camera position and the camera optical axis is given by Equation (68).

[0169]

number

[0170] The calibration unit 130 calculates D or D for all frames within a frame group or for all frames across a frame group that share a common region. 2 The sum of these is calculated, and T0 that minimizes the sum is searched for as the coordinates of the camera position candidate. The calibration unit 130 calculates the focal length f and the rotation matrix R in each frame using the calculated coordinates T0 of the camera position candidate. The formula for calculating the focal length f is derived as follows: 0x , T 0y , T 0z ] T ) is substituted, the equation (69) obtained is rearranged as in equation (70). For the rotation matrix R, R T Since the relationship R = E holds, equation (70) is further transformed into equations (71) to (74) in order. Note that in this example, the parameters s and ρ constituting element matrix A are approximated as s = 0 and ρ = 1, but these parameters may also be set to predetermined values. Noting that equation (74) is a symmetric matrix, six equations related to the focal length f can be obtained as shown in equation (75). By eliminating the coefficient λ while satisfying the relationships shown in each equation in equation (75), equation (76) is obtained.

[0171]

number

[0172]

number

[0173]

number

[0174]

number

[0175]

number

[0176]

number

[0177]

number

[0178]

number

[0179] The calibration unit 130 may calculate the focal length f using any one of the five equations shown in equation (76), but the focal length f calculated for each equation may differ due to the influence of errors related to the processing up to this point. Therefore, the calibration unit 130 may calculate the focal length f calculated from any two to four equations or all five equations in equation (76). 2 Alternatively, the square root of the average value of f may be calculated as the focal length f. The calibration unit 130 can calculate the rotation matrix R according to equations (77) to (82) using the calculated focal length f.

[0180]

number

[0181]

number

[0182]

number

[0183]

number

[0184]

number

[0185]

number

[0186] Appendix 5-3. Bundle adjustment In the process up to Appendix 5-2, the real space projection transformation matrix H g The camera position candidate T0, the focal length f for each frame, and the camera orientation R provisionally calculated based on (a) are algebraically appropriate solutions, but they cannot be said to be likely parameter values ​​in terms of camera geometry. The calibration unit 130 uses the provisionally calculated values ​​as initial values ​​and searches for camera parameters that minimize the reprojection error for known points in real space coordinates. This process is called bundle adjustment. The procedure for bundle adjustment is described below. A known point M on a plane in real space coordinate system i ' and the coordinates of the reprojected points m i The relationship between M and M is given by equations (83) and (84). However, it is assumed that four or more image corresponding points have already been determined in each frame. Note that the aggregated image corresponding points may be located outside the field of view of the image frame, but these may be used in the bundle adjustment process, or only some of them may be used, or only image corresponding points determined within the field of view may be used without using these. In equation (84), M i ' indicates the real space coordinates of the known real space coordinate point detected in frame i. Also, m i is the real space coordinate known point M i ' corresponds to the image correspondence m i The coordinates of are shown.

[0187]

number

[0188]

number

[0189]

number

[0190] The calibration unit 130 defines the error between the image corresponding point and the reprojected point as the reprojection error e, defines the sum of the squared value of the reprojection error shown in Equation (86) for each real-space coordinate known point and each frame in the frame group as the error index, and searches for the coordinate T0 of the camera position, the focal length f, and the rotation matrix R that minimize the error index for some or all of the frames (minimization of the reprojection error). The calibration unit 130 can use the coordinate T0, focal length f, and rotation matrix R of the candidate camera position obtained in the process up to Supplementary Note 5-2 as initial values. Note that the calibration unit 130 may set the coordinate T0 of the camera position obtained for some frames (e.g., the reference frame) as a default value, and use the focal length and rotation matrix R temporarily calculated based on this coordinate T0 as initial values ​​to search for the focal length f and rotation matrix R that minimize the error index for the other frames.

[0191]

number

[0192] Appendix 5-4. Derivation of the projective transformation matrix P between the real space coordinate system and the camera image coordinate system and the real space projective transformation matrix H g , the inter-frame projective transformation matrix H f Update The calibration unit 130 calculates the projective transformation matrix P according to equation (87) using the coordinates T0 of the camera position, the focal length f, and the rotation matrix R optimized in the process of Supplementary Note 5-3. The calibration unit 130 also performs calculations based on equations (88) and (89) to match the optimized parameters and calculates the real space projective transformation matrix H for each frame i. g and update the interframe projection matrix H f may be updated.

[0193]

number

[0194]

number

[0195]

number

[0196] Appendix 5-5. Variability of assumptions In the examples of Supplementary Notes 5-1 to 5-4, it is assumed that the camera parameters constituting element matrix A are s = 0, ρ = 1, and u0 and v0 are default values. However, this is not limiting. Some or all of s, ρ, u0, and v0 may be measured values ​​measured in advance. Furthermore, in the bundle adjustment described in Supplementary Note 5-3, some or all of these parameters may be included in the optimization target as variable variables. If movement of the camera 20 is detected, the camera position T0, which is assumed not to move, may be varied for each frame, and the step described in Supplementary Note 5-2 may be omitted. Furthermore, if no zoom operation is performed, the focal length f, which is assumed to vary for each frame, may be assumed to be constant between frames. The calculation processing unit 120 may determine the assumed conditions to be applied based on operation information acquired from the camera 20, various data indicating the movement state or orientation, or an operation signal indicating an operation on the camera 20 input from the operation input unit 160.

[0197] As described above, the image processing device 10 according to this embodiment calculates an inter-frame projective transformation matrix that indicates the transition of images between frames, identifies image corresponding points corresponding to four or more real-space coordinate points, which are points whose real-space coordinates are known on a real-space plane, in any frame of a frame group consisting of multiple frames, and aggregates and determines image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on the inter-frame projective transformation matrix. The image processing device 10 calculates a real-space projective transformation matrix for the reference frame based on the known real-space coordinate points and the aggregated image corresponding points, calculates real-space projective transformation matrices for the other frames based on the inter-frame projective transformation matrices between the images of the reference frame and the other frames and the real-space projective transformation matrix, determines image corresponding points in the other frames based on the inter-frame projective transformation matrix, and adjusts at least a portion of the camera parameters for each frame so as to reduce the difference between the estimated image corresponding points corresponding to the known real-space coordinate points estimated based on the camera parameters constituting the real-space projective transformation matrix and the image corresponding points corresponding to the known real-space coordinate points.

[0198] According to this configuration, a real space projective transformation matrix indicating the relationship between image corresponding points corresponding to known real space coordinate points in a reference frame, which is one of the multiple frames, and an inter-frame projective transformation matrix indicating image transitions between frames are determined, and the real space projective transformation matrix for each frame is determined based on the determined real space projective transformation matrix and inter-frame projective transformation matrix. Then, at least some of the camera parameters (e.g., focal length) are determined by adjusting the image corresponding points for each frame so that the difference between the image corresponding points estimated from the corresponding known real space coordinate points is reduced. This allows camera calibration to be performed under fewer constraints. For example, camera calibration can be performed for each frame even if markers always set at predetermined known points in real space coordinates do not appear in the captured image. Furthermore, camera calibration estimates camera parameters that reflect changes in the field of view, such as zooming and rotation. This facilitates image analysis across a wider range of shooting conditions.

[0199] This embodiment may be implemented as follows. The image processing device 10 may determine an optical flow indicating the movement of an image for each part starting from the first frame, with the part corresponding to the starting point in the second frame as the end point, and calculate an inter-frame projective transformation matrix for the pair of the first frame and the second frame based on the determined optical flow.

[0200] The image processing device 10 may search for at least some camera parameters that indicate the transformation from the origin to the destination point corresponding to the origin for each set of a predetermined number of origins, and may search for an estimated value of the destination from an inter-frame projective transformation matrix based on the camera parameters searched for the origin, and an inter-frame projective transformation matrix that improves the degree of fit based on the optical flow that matches the destination point to the origin.

[0201] The image processing device 10 may calculate a first inter-frame projective transformation matrix, which is an inter-frame projective transformation matrix for a pair of a first frame and a third frame, which is a frame that is separated from the first frame by a predetermined time or more, calculate a second inter-frame projective transformation matrix, which is an inter-frame projective transformation matrix for a pair of the third frame and a second frame, which is a frame adjacent to the first frame, and calculate an inter-frame projective transformation matrix for the pair of the first frame and the second frame using the first inter-frame projective transformation matrix and the second inter-frame projective transformation matrix.

[0202] The image processing device 10 may execute an inter-frame projective transformation matrix search process to search for inter-frame projective transformation matrices for at least some pairs of adjacent frames such that the inter-frame projective transformation matrix for a pair of the first and last frames in a frame group consisting of multiple frames matches the product of the inter-frame projective transformation matrices for each pair of adjacent frames included in the frame group.

[0203] The image processing device 10 may apply an infinitesimal projection matrix, which is a projection transformation matrix that represents at least some of the variations in the camera parameters, to the inter-frame projection transformation matrix for at least some of the sets of frames included in a frame group consisting of a plurality of frames, to update the inter-frame projection transformation matrix, and may perform an inter-frame projection transformation matrix search process using the updated inter-frame projection transformation matrix.

[0204] The image processing device 10 may calculate, for each pair of frames, a moving average value of the inter-frame projective transformation matrix, calculate a correction value of the inter-frame projective transformation matrix based on the difference between the inter-frame projective transformation matrix and the moving average value, evaluate a first compatibility level that is the compatibility between the inter-frame projective transformation based on the correction value of the inter-frame projective transformation matrix and the optical flow from the origin block to the corresponding block, evaluate a second compatibility level that is the compatibility between the inter-frame projective transformation and the optical flow, and update the inter-frame projective transformation based on the correction value of the inter-frame projective transformation matrix as the inter-frame projective transformation matrix when the first compatibility level is not lower than the second compatibility level.

[0205] The image processing device 10 may define, as a frame group, a plurality of consecutive frames in which the ratio of a common area representing a common field of view is equal to or greater than a predetermined ratio.

[0206] Another aspect of this embodiment may be a program for causing a computer to function as the image processing device 10.

[0207] Another aspect of this embodiment may be an image processing system S1 including a camera 20 that captures an image and an image processing device 10.

[0208] Another aspect of this embodiment may be a method in the image processing device 10, which calculates an inter-frame projective transformation matrix that indicates a transition of images between frames, identifies image corresponding points that correspond to four or more points with known real-space coordinates, which are points with known real-space coordinates on a real-space plane, in any frame of a frame group consisting of multiple frames, aggregates and determines image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculates a real-space projective transformation matrix of the reference frame based on the known real-space coordinates points and the aggregated image corresponding points, calculates a real-space projective transformation matrix of the reference frame based on the inter-frame projective transformation matrix between the images of the reference frame and the other frames and the real-space projective transformation matrix of the reference frame, determines image corresponding points in the other frames based on the inter-frame projective transformation matrix, and determines the camera parameters so as to reduce a difference between the image corresponding points for each frame and estimated values ​​of image corresponding points that are estimated based on the camera parameters from the known real-space coordinate points that correspond to the image corresponding points.

[0209] Note that a portion of the image processing device 10 in the above-described embodiment, such as the arithmetic processing unit 120, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein refers to a computer system built into the image processing device 10, including hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the program may be a program for implementing a portion of the above-described functions, or may be a program that can be implemented in combination with a program already stored in the computer system. Furthermore, part or all of the image processing device 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image processing device 10 may be individually implemented as a processor, or part or all of the blocks may be integrated into a processor. The integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0210] The embodiments of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0211] S1...image processing system, 10...image processing device, 20...camera, 30...display, 120...arithmetic processing unit, 122...image acquisition unit, 124...inter-frame analysis unit, 126...known coordinate point identification unit, 128...real space inter-image analysis unit, 130...calibration unit, 132...output processing unit, 140...storage unit, 150...input / output unit, 160...operation input unit

Claims

1. Calculating an inter-frame projective transformation matrix that indicates the transition of images between frames; Identifying image corresponding points corresponding to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames; aggregating and determining the image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on an inter-frame projective transformation matrix; calculating a real space projection transformation matrix of the reference frame based on the real space coordinate known points and the image corresponding points; calculating a real space projective transformation matrix of the other frame based on an inter-frame projective transformation matrix between the reference frame and the image of the other frame and the real space projective transformation matrix; determining image corresponding points in other frames based on the inter-frame projective transformation matrix; At least a part of the camera parameters is adjusted for each frame so that a difference between an estimated value of an image corresponding point corresponding to the known real space coordinate point, which is estimated based on the camera parameters constituting the real space projection transformation matrix, and the image corresponding point corresponding to the known real space coordinate point is reduced. Image processing device.

2. determining an optical flow indicating an image movement with a part in the second frame corresponding to the part in the first frame as an end point for each part in the first frame as an origin; Calculating the inter-frame projective transformation matrix for the pair of the first frame and the second frame based on the optical flow. The image processing device according to claim 1 .

3. for each of a predetermined number of sets of origin points, finding at least some camera parameters that describe a transformation from the origin points to a corresponding destination point; An estimated value of the end point is estimated from an inter-frame projective transformation matrix based on the camera parameters searched for the origin point, and an inter-frame projective transformation matrix that improves the degree of conformance based on the optical flow that matches the end point with respect to the origin point is searched for. The image processing device according to claim 1 .

4. calculating a first inter-frame projective transformation matrix which is an inter-frame projective transformation matrix relating to a pair of a first frame and a third frame which is a frame separated from the first frame by a predetermined time or more; calculating a second inter-frame projective transformation matrix which is an inter-frame projective transformation matrix relating to a set of the third frame and a second frame which is a frame adjacent to the first frame; Calculating the inter-frame projective transformation matrix for the pair of the first frame and the second frame using the first inter-frame projective transformation matrix and the second inter-frame projective transformation matrix. The image processing device according to claim 1 .

5. An inter-frame projective transformation matrix search process is performed to search for inter-frame projective transformation matrices for at least some of the sets of frames such that the inter-frame projective transformation matrices of the first and last frames of a frame group consisting of a plurality of frames match the product of the inter-frame projective transformation matrices for each set of frames included in the frame group. The image processing device according to claim 1 .

6. updating the inter-frame projective transformation matrices for at least some of the sets of frames included in a frame group consisting of a plurality of frames by applying a minimal projection matrix, which is a projective transformation matrix that represents at least some of the variations in the camera parameters, to the inter-frame projective transformation matrices for at least some of the sets of frames included in the frame group consisting of a plurality of frames; The inter-frame projective transformation matrix search process is executed using the updated inter-frame projective transformation matrix. The image processing device according to claim 5 .

7. For each set of frames Calculating a moving average value of the inter-frame projective transformation matrix; A first degree of conformance is evaluated, which is the degree of conformance between the motion of the image due to the inter-frame projective transformation based on the inter-frame projective transformation matrix and the optical flow. evaluating a second degree of conformance between the motion of the image due to inter-frame projective transformation based on the moving average value and the optical flow; When the second degree of conformance does not decrease below the first degree of conformance, the moving average value is updated as the inter-frame projective transformation matrix. The image processing device according to claim 5 .

8. A plurality of consecutive frames in which the ratio of a common area representing a common field of view is equal to or greater than a predetermined ratio are defined as the frame group. The image processing device according to claim 1 .

9. To the computer A program for causing the image processing device according to claim 1 to function.

10. a camera that captures the image; The image processing device according to claim 1 Image processing system.

11. 1. A method in an image processing device, comprising: Calculating an inter-frame projective transformation matrix that indicates the transition of images between frames; Identifying image corresponding points corresponding to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames; aggregating and determining the image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on an inter-frame projective transformation matrix; calculating a real space projection transformation matrix of the reference frame based on the real space coordinate known points and the image corresponding points; calculating a real space projective transformation matrix of the other frame based on an inter-frame projective transformation matrix between the reference frame and the image of the other frame and the real space projective transformation matrix; determining image corresponding points in other frames based on the inter-frame projective transformation matrix; At least a part of the camera parameters is adjusted for each frame so that a difference between an estimated value of an image corresponding point corresponding to the known real space coordinate point, which is estimated based on the camera parameters constituting the real space projection transformation matrix, and the image corresponding point corresponding to the known real space coordinate point is reduced. Image processing methods.

Images