Image processing device, image processing system, image processing method, and program of same

Abstract

According to the present invention, an inter-frame projective transformation matrix is calculated, image-corresponding points corresponding to four or more real-space-coordinate known points are identified on a real-space plane in any frame of a frame group made up of a plurality of frames, image-corresponding points are aggregated in a reference frame on the basis of the inter-frame projective transformation matrix, a real-space projective transformation matrix of the reference frame is calculated on the basis of the real-space-coordinate known points and the image-corresponding points, a real-space projective transformation matrix of another frame is calculated on the basis of the inter-frame projective transformation matrix between an image of the other frame and the reference frame, an image-corresponding point in the other frame is determined on the basis of the inter-frame projective transformation matrix, and at least a part of camera parameters is adjusted such that difference between an estimated value of the image-corresponding point corresponding to the real-space-coordinate known point estimated from the camera parameters making up the real-space projective transformation matrix for each frame and the image-corresponding point corresponding to the real-space-coordinate known point is reduced.

Description

Image processing device, image processing system, image processing method and program thereof

[0001] This application relates to an image processing device, an image processing system, an image processing method, and a program therefor. This application relates to, for example, camera calibration of captured images. This application claims priority to Japanese Patent Application No. 2024-046899, filed on March 22, 2024, the contents of which are incorporated herein by reference.

[0002] Camera calibration is the process of acquiring the shooting-related parameters necessary for analyzing captured images. These shooting-related parameters are also called camera parameters. Camera parameters include intrinsic and extrinsic parameters. Intrinsic parameters include, for example, the focal length of the lens. Extrinsic parameters include the position and orientation of the camera. Camera calibration can also be considered the process of identifying a projective transformation matrix constructed based on the camera parameters. The projective transformation matrix represents the relationship between the coordinates of an object in the real-space coordinate system and the coordinates of the object as they appear in the camera image coordinate system. The real-space coordinate system is a coordinate system that provides two-dimensional or three-dimensional coordinates indicating any position in real space. The camera image coordinate system is a coordinate system that provides two-dimensional coordinates indicating any position in the captured image captured by the camera. In general, the success and accuracy of video analysis constructed from captured images are affected by camera calibration. Therefore, accurate camera calibration is extremely important in video analysis.

[0003] In conventional camera calibration, the position, orientation, and zoom of the camera are fixed, and a calibration image is captured in advance without changing the angle of view during capture, and a projective transformation matrix is ​​calculated using the calibration image. The calculated projective transformation matrix is ​​then used to analyze a separate image captured thereafter. When capturing the calibration image, the position, orientation, or angle of view of the camera is adjusted so that a predetermined point with known real-space coordinates (referred to herein as a "known real-space coordinate point") is included in the camera's field of view. The projective transformation matrix is ​​calculated using the relationship between the camera image coordinates of the known real-space coordinate point appearing in the captured image and the real-space coordinates in the real space.

[0004] The number of known real space coordinate points used to calculate the projective transformation matrix varies depending on the camera calibration method. For example, in the Direct Linear Transformation (DLT) method described in Non-Patent Document 1, the minimum number of known real space coordinate points required is six. Patent Document 1 proposes a method using three known real space coordinate points by searching for camera internal parameter values ​​using an optimization method.

[0005] JP 2017-116280 A

[0006] Y. I. Abdel-Aziz, H. M. Karara, Direct linear transformation from comparator coordinates to object space coordinates in close-range photogrammetry. ASP symposium on close-range photogrammetry, Falls Church, VA, pp. 1-19, 1971. J. Takamatsu, M. Ae, N. Fujii, Development of the panning DLT method for large measurement ranges. Journal of Physical Education, 42: 19-29, 1997.

[0007] If the camera's angle of view is fixed, the range of real space that appears in the captured image is limited. Therefore, methods have been proposed for analyzing images captured while changing the angle of view. For example, in the panning DLT method described in Non-Patent Document 2, the variable angle of view is limited to pan and tilt, and images are captured while the camera position is fixed using a tripod. Then, in images captured so that a reference point whose real-space coordinates do not change in real space is always included in the field of view, a corresponding point is identified for each frame, and camera calibration is achieved using the camera image coordinates of the identified point. However, the method described in Non-Patent Document 2 requires that the reference point appear in the captured image in each frame. Furthermore, this method cannot accommodate changes in the field of view that accompany changes in the angle of view due to rotation, zoom, etc.

[0008] The present application has been made in consideration of the above points, and aims to provide an image processing device, an image processing system, an image processing method, and a program therefor that can realize camera calibration under fewer constraints.

[0009] (1) The present application has been made to solve the above-mentioned problems, and one aspect of the present application involves calculating an inter-frame projective transformation matrix that indicates a transition of images between frames, identifying image corresponding points that correspond to four or more points with known real-space coordinates, which are points with known real-space coordinates on a real-space plane, in any frame of a frame group consisting of multiple frames, aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculating a real-space projective transformation matrix for the reference frame based on the image corresponding points aggregated with the known real-space coordinate points, calculating a real-space projective transformation matrix for the other frame based on the inter-frame projective transformation matrix between the image of the reference frame and the image of the other frame and the real-space projective transformation matrix, determining image corresponding points in the other frame based on the inter-frame projective transformation matrix, and adjusting at least a portion of the camera parameters for each frame so as to reduce a difference between an estimated value of the image corresponding point corresponding to the known real-space coordinate point, which is estimated based on camera parameters that constitute the real-space projective transformation matrix, and an image corresponding point corresponding to the known real-space coordinate point.

[0010] (2) Another aspect of the present application is a method for an image processing device, the method comprising: calculating an inter-frame projective transformation matrix that indicates a transition of images between frames; identifying image corresponding points that correspond to four or more known real-space coordinate points, which are points whose real-space coordinates are known on a real-space plane, in any frame of a frame group consisting of a plurality of frames; aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix; calculating a real-space projective transformation matrix for the reference frame based on the image corresponding points aggregated with the known real-space coordinate points; calculating a real-space projective transformation matrix for the other frame based on the inter-frame projective transformation matrix between the reference frame and images of the other frame and the real-space projective transformation matrix; determining image corresponding points in the other frame based on the inter-frame projective transformation matrix; and adjusting at least a portion of the camera parameters for each frame so as to reduce a difference between an estimated value of the image corresponding point corresponding to the known real-space coordinate point, which is estimated based on camera parameters that constitute the real-space projective transformation matrix, and the image corresponding point corresponding to the known real-space coordinate point.

[0011] According to the present invention, camera calibration can be achieved under fewer constraints, which makes it easier to analyze images in a wider range of shooting ranges and under a variety of shooting conditions.

[0012] FIG. 1 is an explanatory diagram illustrating an overview of an image processing system according to an embodiment of the present invention. FIG. 2 is a schematic block diagram illustrating an example of the functional configuration of the image processing system according to the embodiment of the present invention. FIG. 3 is an explanatory diagram illustrating optical flow. FIG. 4 is a first explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. FIG. 5 is a second explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. FIG. 6 is a sixth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. FIG. 7 is a seventh explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. FIG. 8 is an eighth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. FIG. 9 is a ninth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. FIG. 10 is an explanatory diagram illustrating a relationship between inter-frame projective transformation and real space inter-image projective transformation. FIG. 11 is an explanatory diagram illustrating bundle adjustment. 1 is an explanatory diagram illustrating an example of a relationship between an error in a camera position and a focal length; FIG. 2 is a flowchart illustrating an example of an image processing method according to an embodiment; FIG. 3 is a flowchart illustrating a first example of a calculation process of an inter-frame projective transformation matrix; FIG. 4 is a flowchart illustrating a smoothing process of an inter-frame projective transformation matrix;

[0013] Hereinafter, an embodiment of the present application will be described with reference to the drawings. FIG. 1 is an explanatory diagram illustrating an overview of an image processing system S1 according to this embodiment. The image processing system S1 includes an image processing device 10 and a camera 20. The image processing device 10 acquires image data representing images captured at different times for each frame (sometimes referred to herein as "captured images") from the camera 20. The image processing device 10 performs calibration on the image represented by the acquired image data and estimates a homography matrix for each frame based on parameters of the camera 20 (sometimes referred to herein as "camera parameters"). The homography matrix indicates the correspondence between a coordinate system set in a three-dimensional real space to be captured (sometimes referred to herein as a "real space coordinate system") and a two-dimensional coordinate system set in the captured image (sometimes referred to herein as a "camera image coordinate system" or "image coordinate system"). In the example of FIG. 3, a vector m representing coordinates (u, v) in the captured image ip corresponds to the product of a vector M representing coordinates (X, Y, Z) in real space, a homography matrix P, and a scale factor k. The projective transformation matrix P is uniquely determined based on the camera parameters. The image processing device 10 analyzes the image for each frame based on the estimated projective transformation matrix.

[0014] In the calibration according to this embodiment, the image processing device 10 calculates an inter-frame projective transformation matrix that indicates the transition of images between frames. In any frame of a frame group consisting of images of multiple frames, the image processing device 10 identifies image corresponding points corresponding to each of four or more points whose real-space coordinates are known on a real-space plane. The image processing device 10 determines one reference frame from the frame group as a reference, and determines image corresponding points in the reference frame that correspond to image corresponding points in the arbitrary frame based on an inter-frame projective transformation matrix between the arbitrary frame in which image corresponding points have been identified and the reference frame. The image processing device 10 calculates a real-space projective transformation matrix based on the correspondence between points with known real-space coordinates and image corresponding points in the reference frame. The image processing device 10 calculates a real-space projective transformation matrix for another frame based on the inter-frame projective transformation matrix between the reference frame and the other frame and the real-space projective transformation matrix for the reference frame. The image processing device 10 determines image corresponding points in the other frame that correspond to image corresponding points in the reference frame based on the inter-frame projective transformation matrix between the reference frame and the other frame. The image processing device 10 determines, for each frame, the difference between the estimated value of the image corresponding point estimated based on the camera parameters constituting the real-space projective transformation matrix and the coordinate of the image corresponding point corresponding to the real-time coordinate known point as a reprojection error, and adjusts some or all of the camera parameters so as to minimize the reprojection error. The image processing device 10 analyzes the captured image of that frame using the camera parameters determined for that frame.

[0015] The camera 20 captures images representing scenes of subjects within its field of view at different times. The camera 20 is, for example, a digital video camera. A digital video camera has the function of continuously capturing moving images. Generally, moving images are composed of still images for each frame captured at different times, and are captured or displayed at a predetermined frame rate (for example, 30 to 300 fps). The camera 20 transmits image data representing the captured images to the image processing device 10.

[0016] The image processing system S1 realizes calibration for each frame of a captured image, allowing for changes over time in the orientation and zoom of the camera 20. However, it is necessary to set four or more feature points as known real-space coordinates on a plane that spreads over a relatively wide area in real space, and to set the coordinates of each of these points in advance in the image processing device 10. The feature points are artificial or natural objects that have morphological characteristics that make them easily distinguishable from other objects or patterns.

[0017] The image processing system S1 according to this embodiment is applied to captured images obtained by continuously capturing moving subjects from a distance, for example, at a stadium or other event venue. In the example of FIG. 1 , a scene of an athlete running on a track Tr of an athletics stadium is captured, and hurdle placement marks Mk on the track Tr are used as feature points used for known real-space coordinate points. The feature points do not necessarily need to be captured in advance. Four or more feature points do not necessarily need to be captured in all frames of the image; they need only be present in at least some of the frames of a group of frames to be processed at one time. For example, only three or fewer feature points may appear in any one of the frames, or no feature points may appear in some of the frames. In this way, marks used in events such as competitions, or patterns or objects that do not hinder the movement of the subject and are not continuously obscured by the subject or other objects, can be used as known real-space coordinate points. The feature points may also be used as references for the real-space coordinate system. For example, a horizontal plane on which multiple feature points are located may be defined as a plane whose coordinate value in the vertical direction (z direction) is zero, or one of those feature points or the center of gravity may be defined as the origin of the real space coordinate system.

[0018] The position and orientation of the camera 20 do not need to be fixed using a device such as a tripod. Changes in either or both of the position and orientation of the camera 20 are permitted. Changes in the orientation of the camera 20 can be panned, tilted, or rotated. For example, the image processing device 10 according to this embodiment may also process images captured by a photographer holding the camera 20 while moving and adjusting the zoom to follow the movement of the subject. Therefore, even in a large space with a side or diameter of the capture range exceeding 100 m, it is possible to process images captured using zooming or other camerawork, as long as known real-space coordinate points are set in the space.

[0019] Next, an example of the functional configuration of the image processing system S1 according to this embodiment will be described. Fig. 2 is a schematic block diagram showing an example of the functional configuration of the image processing system S1 according to this embodiment. In the example of Fig. 2, the image processing system S1 includes an image processing device 10, a camera 20, and a display 30. The image processing device 10 includes an arithmetic processing unit 120, a storage unit 140, an input / output unit 150, and an operation input unit 160. The image processing device 10 may be configured as a general-purpose information processing device such as a personal computer or a workstation, or may be configured as part of a dedicated image production device or image editing device.

[0020] The arithmetic processing unit 120 performs processing and control to realize various functions of the image processing device 10. The arithmetic processing unit 120 may be configured using dedicated components, or may be configured as a general-purpose computer system including a processor and a storage medium. The processor may read a predetermined program stored in advance in a storage medium and execute the read program to realize the functions of each functional unit in cooperation with the storage medium and other hardware. In this application, "executing a program" refers to executing processing instructed by instructions written in the program. The processor includes, for example, a CPU (Central Processing Unit). The processor may also include a different type of arithmetic processing device, such as a GPU (Graphics Processing Unit). Each functional unit of the arithmetic processing unit 120 will be described later.

[0021] The storage unit 140 stores various data used in processing in the arithmetic processing unit 120 and various data acquired or generated by the arithmetic processing unit 120. The storage unit 140 includes an auxiliary storage device such as a hard disk drive (HDD) or a solid-state storage device (SSD). The input / output unit 150 inputs and outputs various data to and from other devices via a wired or wireless connection. The input / output unit 150 outputs image data input from the camera 20 to the arithmetic processing unit 120. The input / output unit 150 outputs display data input from the arithmetic processing unit 120 to the display 30. The input / output unit 150 includes, for example, an input / output interface. The input / output unit 150 may be connected to other devices via a communication network so that data can be transmitted and received. The operation input unit 160 receives user operations and generates operation signals in response to the received operations. The operation input unit 160 outputs the generated operation signals to the arithmetic processing unit 120. The operation input unit 160 may have, for example, a general-purpose input device such as a mouse or a touch sensor, or may have a dedicated input device such as a button, a lever, or a dial.

[0022] The display 30 displays a display screen indicated by display data input from the image processing device 10. The display 30 may be any type of display monitor, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display.

[0023] One or both of the camera 20 and the display 30 may be configured separately from the image processing device 10, as exemplified in Fig. 2, or may be configured integrally with the image processing device 10. Furthermore, the operation input unit 160 may be configured integrally with the image processing device 10, as exemplified in Fig. 2, or may be configured separately. The touch sensor and the display 30 constituting the operation input unit 160 may be configured separately, or may be integrated into a touch panel.

[0024] Next, a description will be given of an example of the functional configuration of the arithmetic processing unit 120. The arithmetic processing unit 120 includes an image acquisition unit 122, an inter-frame analysis unit 124, a known coordinate point identification unit 126, a real-space inter-image analysis unit 128, a calibration unit 130, and an output processing unit 132.

[0025] The image acquisition unit 122 receives image data input from the camera 20 using the input / output unit 150. The image acquisition unit 122 sequentially stores the acquired image data in the storage unit 140. The storage unit 140 accumulates image data acquired at different times. The inter-frame analysis unit 124 calculates an inter-frame projective transformation matrix that indicates the transition of images between frames acquired at different times. The inter-frame analysis unit 124 reads image data to be processed from the storage unit 140 and calculates optical flow, which indicates the movement from the image of the current frame to the image of another frame shown in the read image data. The optical flow is expressed by a motion vector that indicates the amount and direction of movement for each part of the image. The optical flow is calculated so that the pattern at a certain part (starting point) in the current frame is as similar as possible to the pattern at a part (end point) after movement in another frame.

[0026] As shown in FIG. 4, the inter-frame analysis unit 124 calculates the coordinates [u, v, 1] of a certain part in the current frame i. T (where T denotes the transpose of a vector or matrix), the pattern at this coordinate is calculated by the inter-frame projection transformation matrix H i→i+1 f The corresponding coordinates after movement in another frame i+1 indicated by the coordinate values ​​obtained by multiplying [u', v', 1] T The inter-frame projection transformation matrix H i→i+1 f However, in Figure 4, h 00 ,…,h 21 ,1 is the inter-frame projective transformation matrix H i→i+1 f where k denotes an element of the inter-frame projective transformation matrix. Furthermore, k is a scale factor that expresses the arbitrariness of the constant multiple of the inter-frame projective transformation matrix. The inter-frame analysis unit 124 searches for the inter-frame projective transformation matrix by, for example, performing a procedure such as regressing the optical flow calculated between frames. The inter-frame analysis unit 124 can also calculate inter-frame projective transformation matrices between other pairs of frames. For example, the inter-frame analysis unit 124 sequentially calculates the inter-frame projective transformation matrix from one frame to the next frame for each frame. The inter-frame analysis unit 124 stores the inter-frame projective transformation matrix calculated for each pair of frames in the storage unit 140.

[0027] The inter-frame analysis unit 124 can use known techniques (e.g., block matching, feature point correspondence, etc.) when calculating the optical flow. Block matching calculates the optical flow for each block that constitutes a portion of the current frame and is composed of a predetermined number of pixels. FIG. 5 shows an example of the optical flow Of calculated by block matching. In the example of FIG. 5 , the optical flow from each starting block, which is divided into regular intervals by a grid and has a plurality of pixels arranged therein, to the end block is shown. Block matching searches for the end block whose pattern most closely matches the pattern represented in the starting block, and whose matching degree is equal to or exceeds a predetermined reference value.

[0028] In the image illustrated in FIG. 5 , accurate optical flow may not be calculated in or around areas where non-planar surfaces appear. Furthermore, even in areas where planar surfaces appear, accurate optical flow may not be calculated for areas with constant brightness or color (monotone) or areas that display patterns that are spatially repeated at regular intervals. These optical flows, including errors, can cause an inaccurate estimation of the inter-frame projective transformation matrix. To reduce these effects, the inter-frame analysis unit 124 may determine the inter-frame projective transformation matrix by performing optical flow regression using a robust method. The inter-frame analysis unit 124 may use, for example, the RANSAC (Random Sample Consensus) method. Specific examples of methods for calculating the inter-frame projective transformation matrix will be described later as Methods 1 to 7.

[0029] Returning to FIG. 2, for each of four or more known real-space coordinates in the real space represented in the image, the known real-space coordinate identification unit 126 identifies the part where the known real-space coordinate appears as an image corresponding point (digitization of the known real-space coordinate). The known real-space coordinate identification unit 126 causes, for example, the display 30 to display a display screen on which at least one frame of an image to be analyzed (sometimes referred to as a “target image” in the present application) is arranged for the output processing unit 132. The known real-space coordinate identification unit 126 identifies the part within the target image indicated by the operation signal input from the operation input unit 160 as an image corresponding point. Thereby, while viewing the display screen displayed on the display 30, the user can recognize the feature points where the known real-space coordinates are set and indicate the part where the feature points appear by an operation. In the image of the track Tr illustrated in FIG. 1, the position of the mark Mk is indicated as an image corresponding point. The known real-space coordinates to be the target of identification of the image corresponding point only need to have four or more appearing in an arbitrary frame in the group of frames processed at one time. The four or more different known real-space coordinates may all be represented in one frame or may be distributed over two or more frames. The known real-space coordinate identification unit 126 stores in the storage unit 140 known real-space coordinate information indicating the frames and the coordinates of the image corresponding points specified for each known real-space coordinate.

[0030] The real-space image-to-image analysis unit 128 determines one of the frames in the group of frames to be processed as a reference frame. The real-space image-to-image analysis unit 128 can determine, as the reference frame, for example, the first frame, the central frame, or the last frame in the group of frames to be processed, or the frame in which the coordinates of the most image corresponding points for the known real-space coordinates indicated in the known real-space coordinate information stored in the storage unit 140 are specified. The real-space image-to-image analysis unit 128 determines the inter-frame projective transformation matrix from the frame in which the image corresponding points of the known real-space coordinates are specified to the reference frame using the inter-frame projective transformation matrix stored in the storage unit 140. The real-space image-to-image analysis unit 128 determines the inter-frame projective transformation matrix H from the frame j (j < i) preceding the reference frame i to the reference frame i as H j→i f as Hi-1→i f …H j+1→j+2 f H j→j+1 f The real space inter-image analysis unit 128 calculates an inter-frame projective transformation matrix H j→i f (H i→i+1 f ) -1 …(H j-2→j-1 f ) -1 (H j-1→j f ) -1 The real-space inter-image analysis unit 128 can determine the coordinates of the image corresponding points in the reference frame by multiplying the coordinates of the image corresponding points identified in that frame by the determined inter-frame projective transformation matrix. The image corresponding points are aggregated in the reference frame, and the coordinates of the image corresponding points in the reference frame are determined for each of the four or more real-space coordinate known points.

[0031] The real space inter-image analysis unit 128 calculates a real space projection transformation matrix H for transforming an arbitrary point M′ on a plane in the real space coordinate system to a point m in the camera image coordinate system based on the real space coordinates and the image corresponding points for each real space coordinate known point in the reference frame k. k g Calculate the real space projection transformation matrix H g The calculation method of will be described later in Supplementary Note 4. The real space inter-image analysis unit 128 calculates the real space projection transformation matrix H k g The inter-frame projective transformation matrix H from the reference frame to the other frame is k→j f and multiplying it by , the real space projection transformation matrix H j g FIG. 15 shows the real space projection transformation matrix H i-1 g Then, the inter-frame projective transformation matrix H i f The matrix H obtained by multiplying i f Hi-1 g is the real space projection transformation matrix H i g In addition, the real space projection transformation matrix H i g and the inter-frame projective transformation matrix H i+1 f By multiplying by , the real space projection transformation matrix H i+1 g The real space inter-image analysis unit 128 stores the real space projective transformation matrix calculated for each frame in the storage unit 140. The real space projective transformation matrix will be described later in Supplementary Note 4.

[0032] The calibration unit 130 adjusts the camera parameters (bundle adjustment) so that image corresponding points in the camera image coordinate system corresponding to points with known real-space coordinates in each frame closely match estimated values ​​of image corresponding points transformed from the known real-space coordinate points in that frame using camera parameters constituting a real-space projective transformation matrix. The camera parameters to be adjusted may be limited to some element parameters (e.g., focal length), and other element parameters may be fixed to constant values. The calibration unit 130 determines image corresponding points in each of the other frames using an inter-frame projective transformation matrix for each image corresponding point in the reference frame. The calibration unit 130 sets the camera parameters related to the real-space projective transformation matrix as initial values ​​and calculates estimated values ​​of image corresponding points estimated for points with known real-space coordinates using the camera parameters. The calibration unit 130 defines the difference between the image corresponding points and their estimated values ​​as a reprojection error and searches for camera parameters that minimize the reprojection error. FIG. 16 illustrates a case in which the difference in coordinate values ​​between an image corresponding point m and its estimated value m' is defined as a reprojection error, and the camera parameters are adjusted so as to reduce the reprojection error. The calibration unit 130 stores the camera parameters determined for each frame in the storage unit 140. The camera parameters will be described later in Supplementary Note 5. In particular, bundle adjustment will be described in Supplementary Note 5-3. The calibration unit 130 may analyze the image of that frame using the camera parameters determined for each frame. The calibration unit 130 stores corrected image data indicating the corrected image in the storage unit 140.

[0033] The output processing unit 132 outputs various data to the outside of the image processing device 10. The output processing unit 132 may configure a display screen and output display data indicating the configured display screen to the display 30 to display the display screen. The output processing unit 132 may analyze or correct an acquired image using the camera parameters obtained by the calibration unit 130. The output processing unit 132 may output display data indicating the corrected image to the display 30 to display the corrected image. When the output processing unit 132 receives a corrected image request command from another device via the input / output unit 150, it may transmit corrected image data indicating the corrected image to the other device that sent the command. The output processing unit 132 may output display data indicating the analysis result to the display 30 to display the analysis result. When the output processing unit 132 receives an analysis command from another device via the input / output unit 150, it may transmit analysis data indicating the analysis result to the other device that sent the command or to another device.

[0034] Next, an example of an image processing method according to this embodiment will be described. FIG. 18 is a flowchart illustrating the image processing method according to this embodiment. (Step S102) The image acquisition unit 122 of the image processing device 10 receives image data representing a captured image from the camera 20 and sequentially stores it in the storage unit 140. (Step S104) The inter-frame analysis unit 124 calculates an inter-frame projective transformation matrix representing the movement of the image between different frames. (Step S106) The known-coordinate point identification unit 126 displays the target image to be processed on the display 30 and identifies image corresponding points for each of four or more real-space coordinate known points based on coordinates specified by operation signals input from the operation input unit 160. (Step S108) The real-space inter-image analysis unit 128 determines image corresponding points in the image in the reference frame using the image corresponding points for the real-space coordinate known points identified in an arbitrary frame and the inter-frame projective transformation matrix. The real-space inter-image analysis unit 128 calculates a real-space projective transformation matrix for the reference frame based on the correspondence between the real-space coordinates of the known real-space coordinate points and the coordinates of the image corresponding points in the reference frame. The real-space inter-image analysis unit 128 calculates the real-space projective transformation matrix for each frame using the inter-frame projective transformation matrix for each frame pair and the real-space projective transformation matrix for the reference frame.

[0035] (Step S110) The calibration unit 130 determines image corresponding points in each frame using an inter-frame projective transformation matrix for each image corresponding point in the reference frame. The calibration unit 130 sets camera parameters related to the real-space projective transformation matrix as initial values ​​and calculates estimated values ​​of image corresponding points estimated using the camera parameters for points with known real-space coordinates. The calibration unit 130 defines the difference in coordinate values ​​between the image corresponding points and their estimated values ​​as the reprojection error, and searches for some or all of the camera parameters to minimize the reprojection error (minimizing the reprojection error). (Step S112) The output processing unit 132 executes processing using the obtained camera parameters. For example, the output processing unit 132 corrects the image of the frame using the obtained camera parameters. The output processing unit 132 outputs display data representing a display screen including the corrected image to the display 30, causing the display screen to be displayed. The output processing unit 132 transmits the obtained camera parameters to another device, for example, via the input / output unit 150. The processing of FIG. 18 then ends.

[0036] Next, a specific example of a method for calculating the inter-frame projective transformation matrix according to this embodiment will be described. Method 1: The inter-frame analysis unit 124 may ignore the optical flow calculated in a region where a moving object appears in one or both of the image of the starting frame (hereinafter sometimes referred to as the "starting frame", e.g., the current frame) and the image of the ending frame (hereinafter sometimes referred to as the "ending frame", e.g., the next frame), and may calculate the inter-frame projective transformation matrix using the optical flow of other regions. The inter-frame analysis unit 124 may also calculate the inter-frame projective transformation matrix by applying a robust regression method to the optical flow of other regions.

[0037] In the example of FIG. 5 , optical flows originating from the area in the current frame where a running athlete appears and the area within a certain width from the outer edge are ignored. The inter-frame analysis unit 124 can estimate the area in the target image where a moving object appears. The known image processing technology may be an image recognition technology that recognizes an image area of ​​a moving object, such as a human, or the following method may be used. For example, the inter-frame analysis unit 124 calculates the distribution of the number of optical flows for each pair of movement amount and movement direction for the image of the start frame, and determines the pair of movement amount and movement direction with the largest number as the mode of the optical flow. The inter-frame analysis unit 124 estimates the area associated with the optical flow whose deviation from the mode exceeds a certain value as the area where a moving object appears. This is because optical flows whose deviation from the mode is within a certain value are considered to represent the overall movement of the entire image from the start frame to the end frame. For example, when the degree of correspondence between a pattern represented in the peripheral area of ​​the optical flow start point in the start frame and a pattern represented in the peripheral area of ​​the optical flow end point in the end frame does not meet a predetermined reference value, the inter-frame analysis unit 124 estimates the area related to the optical flow as an area where a moving object appears. These methods can reduce the processing load compared to image recognition technology.

[0038] Method 2: When calculating an inter-frame projective transformation matrix between a source frame and a temporally adjacent destination frame (for example, between adjacent frames), the inter-frame analysis unit 124 may calculate the product of the inter-frame projective transformation matrix from the source frame to the reference frame and the inter-frame transformation matrix from the reference frame to the destination frame as the inter-frame projective transformation matrix from the source frame to the destination frame. The reference frame is a frame that is temporally further away from the source frame and the destination frame than the interval between them. In the example of FIG. 6 , the inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H from the current frame i to the reference frame i+a. i→i+a f and the inter-frame projective transformation matrix H from the reference frame i+a to the next frame i+1. i+a→i+1 f Product Hi+a→i+1 f H i→i+a f is the inter-frame projective transformation matrix H i→i+1 f The calculation is performed as follows. Because the deformation and movement of a moving object are relatively small between temporally adjacent frames, it may be difficult to estimate the area in which the object appears. If the optical flow associated with the moving object cannot be excluded when estimating the inter-frame projective transformation matrix, an error will occur in the calculated inter-frame projective transformation matrix. By deriving the inter-frame projective transformation matrix between temporally adjacent frames from the inter-frame projective transformation matrix between frames with a longer time interval, as in this method, the influence of the error can be reduced. Note that reference frame a may be subsequent to the end frame or prior to the start frame. Reference frame a may be, for example, the start or end frame of a group of frames (described below).

[0039] Method 3: The inter-frame analysis unit 124 may determine the inter-frame projective transformation matrix by restricting the degrees of freedom of changes in camera parameters between frames. The inter-frame analysis unit 124 may also adjust the degrees of freedom depending on the operation status of the camera 20. The inter-frame projective transformation matrix is ​​a 3-row, 3-column matrix with 8 degrees of freedom that is uniquely determined based on the camera parameters. In other words, the degrees of freedom of the inter-frame projective transformation matrix correspond to the degrees of freedom of the camera parameters. Restricting the degrees of freedom of the camera parameters means assuming that only some camera parameters change and ignoring changes in other element parameters. The camera parameters include position (3 degrees of freedom), orientation (3 degrees of freedom), and focal length (1 degree of freedom).

[0040] For example, if the movement distance of the camera 20 between frames is equal to or less than a predetermined reference value, the inter-frame analysis unit 124 can determine an inter-frame projective transformation matrix based on a model with four degrees of freedom in which the orientation and focal length are variable, assuming that the position is constant. The inter-frame analysis unit 124 can estimate the movement distance by, for example, acquiring signals indicating the movement state of the camera detected by position, speed, and acceleration sensors provided in the camera 20, and processing the acquired signals together with the time and time interval of each frame.

[0041] For example, if the range of real space captured by camera 20 is sufficiently far from the position of camera 20 and it is determined that the change in position of camera 20 is relatively small compared to the range of real space captured in the image, the inter-frame analysis unit 124 can determine an inter-frame projective transformation matrix based on a model with four degrees of freedom in which the orientation and focal length are variable, under the assumption that the position is constant.

[0042] When no zoom operation is performed, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix by assuming that the focal length is constant and varying only the other parameters. When the position of the camera 20 can be assumed to be constant and no zoom operation is performed, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix based on a three-degree-of-freedom model in which the camera position and focal length are fixed and only the orientation is variable. The inter-frame analysis unit 124 can determine whether a zoom operation is performed, for example, based on whether an operation signal indicating a zoom command for the camera 20 is input. By restricting the degrees of freedom of the camera parameters as described above, the risk of calculating an unrealistic inter-frame projective transformation matrix from an optical flow that contains a significant amount of error is reduced, and the inter-frame projective transformation matrix is ​​calculated stably and with a certain degree of accuracy. Note that the restriction on the degrees of freedom of the camera parameters will be described later in Supplementary Note 1.

[0043] The inter-frame projective transformation matrices between adjacent frames calculated individually as described above can also be used to calculate an inter-frame projective transformation matrix between temporally separated frames by multiplying them together. However, the inter-frame projective transformation matrix between temporally separated frames calculated in this manner accumulates errors contained in the inter-frame projective transformation matrices between the individual adjacent frames. The inter-frame analysis unit 124 can reduce the accumulated errors by applying any one or a combination of the processes related to the following methods 4 to 7.

[0044] Method 4: The inter-frame analysis unit 124 may search for an inter-frame projective transformation matrix for each pair of frames in a group of frames, such that the inter-frame projective transformation matrix for each pair of frames in the group of frames is highly consistent with the inter-frame projective transformation matrix obtained by multiplying the optical flow between any two frames (e.g., the first frame to the last frame of the group of frames) by the inter-frame projective transformation matrix for the other frames. In the example of Figure 7, the total product of the inter-frame projective transformation matrix from the first frame i to the last frame i+l of the group of frames and the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames is used as an estimate of the inter-frame projective transformation matrix from the first frame i to the last frame i+l, and the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames is searched for while considering the high consistency between this estimate and the optical flow between the two frames.

[0045] The frame section to be evaluated for compatibility is not limited to a section starting from the start frame and ending at the last frame. This section may be any section bounded by two different frames within a frame group. FIG. 8 illustrates an example in which the product of the sum of the inverse matrices of the inter-frame projective transformation matrices for each pair of adjacent frames from the start frame i to the penultimate frame i+l-1 and the inter-frame projective transformation matrix from the start frame i to the final frame l is used as the estimated value for the inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l. The inter-frame analysis unit 124 searches for the inter-frame projective transformation matrix for each pair of adjacent frames from the start frame i to the penultimate frame i+l-1 in the frame group and the inter-frame projective transformation matrix from the start frame i to the final frame l, while taking into account the need to increase the compatibility between the estimated inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l and the optical flow between the two frames.

[0046] Assuming that the inter-frame projective transformation matrices for some frame pairs are default, the inter-frame analysis unit 124 may perform the above inter-frame projective transformation matrix search only for the remaining frame pairs, and leave the inter-frame projective transformation matrices for some frame pairs fixed to default values ​​and exclude them from the inter-frame projective transformation matrix search. In the example of Figure 9, similar to the example of Figure 7, the total product of the inter-frame projective transformation matrices for each pair of adjacent frames in the frame group is used as an estimate of the inter-frame projective transformation matrix from the starting frame i to the final frame i+l, and the inter-frame projective transformation matrix for each pair of adjacent frames is searched for so as to achieve a higher degree of compatibility with the optical flow between the two frames. Here, the inter-frame analysis unit 124 only searches for the inter-frame projective transformation matrix from frame i2 to frame i2+1, and considers the inter-frame projective transformation matrices for the other frame pairs to be default values.

[0047] 10 , similar to the example of FIG. 8 , the inter-frame analysis unit 124 searches for an inter-frame projective transformation matrix for each pair of adjacent frames from the starting frame i to the penultimate frame i+l-1 in the frame group, and an inter-frame projective transformation matrix from the starting frame i to the final frame l, so as to achieve a higher degree of conformity between the estimated inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l and the optical flow between the two frames. Here, the inter-frame analysis unit 124 searches only for the inter-frame projective transformation matrix from frame i2 to frame i2+1, and considers the inter-frame projective transformation matrices for each pair of other frames to be default values. For example, a generally accurate inter-frame projective transformation matrix (i.e., an inter-frame projective transformation matrix obtained with a degree of accuracy that is expected to be accurate in advance) is used as the default inter-frame projective transformation matrix.

[0048] In Method 4, the longer the period from the start frame i to the final frame i+l, the greater the effect of reducing cumulative error. However, to calculate an optical flow that can accurately estimate the inter-frame projective transformation matrix between both frames, the ratio (hereinafter referred to as the "image overlap rate") of the area in the image where a common area in real space appears between both frames (hereinafter referred to as the "common area") must be at least a certain level. Therefore, the inter-frame analysis unit 124 identifies the common area retrospectively based on the inter-frame projective transformation matrix searched for each pair of different frames. The inter-frame analysis unit 124 may define a series of frames as a single interval in which the identified common area occupies a certain ratio or more. The image overlap rate of the common area is set in advance based on a balance with the influence of errors contained in the optical flow. A series of frames consisting of multiple frames in which a common scene appears is determined based on the image overlap rate of the common area. The inter-frame analysis unit 124 may execute Method 4 for each predetermined frame group, or may perform the search for the inter-frame projective transformation matrix and the setting of the frame groups in parallel. Note that common frames may exist among multiple frame groups. In this case, overlapping of time series between the frame groups is permitted. By executing Method 4 for each frame group, the inter-frame analysis unit 124 can reduce accumulated errors across multiple frame groups.

[0049] Method 5: The inter-frame analysis unit 124 may perform a micro-transformation on the inter-frame projective transformation matrix of some or all of the frame pairs in the frame group, and then check the compatibility of the transformed inter-frame projective transformation matrix with the optical flow according to Method 4 to search for an inter-frame projective transformation matrix with higher compatibility. The projective transformation matrix is ​​expressed as the product of element matrices indicating the components of each element parameter (see Supplementary Note 1). Element parameters refer to individual parameters that are elements of the camera parameters. Examples of element parameters include the position, orientation, and focal length of the camera 20. In the micro-transformation, the inter-frame analysis unit 124 may assume that some or all of the element parameters have randomly and minutely fluctuated, and calculate the product of element matrices corresponding to the amount of variation associated with that variation as the micro-transformation matrix ΔH.

[0050] 11 illustrates a case in which, in the method illustrated in FIG. 7 , each interframe projection transformation matrix between adjacent frames is multiplied by an infinitesimal transformation matrix to search for an interframe projection transformation matrix that improves the sum of the compatibility between the interframe projection transformation matrix and the optical flow between adjacent frames and the compatibility between the interframe projection transformation matrix and the optical flow between the start frame and the final frame. Here, the interframe analysis unit 124 compares compatibility 1 obtained using the interframe projection transformation matrix before transformation with compatibility 2 obtained using the interframe projection transformation matrix after transformation. If compatibility 2 is higher than compatibility 1, the interframe projection transformation matrix is ​​updated to the post-transformation interframe projection transformation matrix as the new interframe projection transformation matrix. Otherwise, the interframe projection transformation matrix is ​​not updated. The interframe analysis unit 124 then repeats the infinitesimal transformation of the interframe projection transformation matrix, comparing the compatibility before and after transformation, and updating the interframe projection transformation matrix until the compatibility no longer improves or a predetermined number of repetitions is reached. When roughly accurate interframe projective transformation matrices are obtained between all adjacent frames in a frame group, it may be difficult to further reduce the accumulated error that appears in the interframe projective transformation matrices between temporally distant frames, even by repeatedly using a robust regression method such as the RANSAC method. Even in such cases, adding random small transformations can further search for interframe projective transformation matrices that would not be identified as candidates by the above method. This provides an opportunity to reduce the accumulated error.

[0051] In addition, when an ideal value (or target value) of the inter-frame projective transformation matrix between two frames in a certain frame group is set or known, the inter-frame analysis unit 124 may make small changes to the inter-frame projective transformation matrix between other frames based on the ideal value to check whether the degree of conformance with the optical flow improves. Here, the inter-frame analysis unit 124 calculates the difference H between the ideal value and the current value. diffThe inter-frame analysis unit 124 calculates the difference component dH by applying the difference component dH distributed to each frame pair to correct the inter-frame projective transformation matrix (see FIG. 12 ). The inter-frame analysis unit 124 then determines whether the corrected inter-frame projective transformation matrix improves the degree of conformance with the optical flow for the frame group compared to the degree of conformance based on the inter-frame projective transformation matrix before correction. If the inter-frame analysis unit 124 determines that the degree of conformance will improve, it updates the inter-frame projective transformation matrix before correction to the corrected one; if it determines that the degree of conformance will not improve, it does not update the inter-frame projective transformation matrix before correction to the corrected one. Note that minute transformation will be described later in Supplementary Note 2.

[0052] Method 6: For each pair of adjacent frames to be processed, the inter-frame analysis unit 124 may calculate a weighted average of inter-frame projective transformation matrices between multiple frames that are consecutive in time series, with the target frames as the reference. The inter-frame analysis unit 124 calculates the compatibility between the inter-frame projective transformation matrix and the optical flow before and after calculating the weighted average. If the sum of the calculated compatibility between a pair of frames does not decrease due to the weighted average, the inter-frame analysis unit 124 updates the weighted average as a new inter-frame projective transformation matrix. If the sum of the calculated compatibility decreases due to the weighted average, the inter-frame analysis unit 124 adopts the inter-frame projective transformation matrix before calculating the weighted average. This allows the inter-frame analysis unit 124 to accurately estimate image motion while reducing fluctuations in the projective transformation matrix between frames. The weighted average will be described later in Supplementary Note 3.

[0053] Method 7: The inter-frame analysis unit 124 may determine whether a common region representing a common real space is shared even between frames that are not consecutive in time, and may perform any of methods 4 to 6 between frames that share a common region. Fig. 13 shows a case where a group of frames (shown as three rectangular frames on the left and one rectangular frame on the right in Fig. 13) that share a common region are generated at intervals in time, and the inter-frame projective transformation matrices of the intermittent frame group that intermittently connects these frames are calculated in chronological order as H0 f ,… , H4f The inter-frame analysis unit 124 calculates the total product H intermit f may be replaced with the total product of the inter-frame projective transformation matrices between all adjacent frames during this period, and any one of methods 4 to 6 may be performed. After performing a process to calculate or update the inter-frame projective transformation matrices between frames included in the intermittent frame group, the inter-frame analysis unit 124 may set the inter-frame projective transformation matrix obtained by performing this process as a default value and perform any one of methods 4 to 6 for all adjacent frames included between each of the frames (see FIG. 14 ).

[0054] Before processing intermittent frame groups, the inter-frame analysis unit 124 may perform any of methods 4 to 6 on individual consecutive frame groups including those frames, as illustrated in FIGS. 13 and 14 . When determining frames that share a common area, the inter-frame analysis unit 124 may, for example, determine a frame in a frame group whose common area is shared with another frame at a ratio equal to or greater than a predetermined ratio as a frame that shares a common area with the frame group. When capturing an image, a user may point the camera 20 from one real space region to another real space region and then re-point it back toward the original real space region. In such a case, multiple groups of temporally separated frames representing the original real space region are generated, resulting in cumulative error in the inter-frame projective transformation matrix connecting the frames. This method can reduce cumulative error between non-consecutive frames that represent a common real space region.

[0055] The inter-frame analysis unit 124 may execute any one of the processes of methods 1 to 7, or may execute a combination of some or all of them as long as no contradictions arise. The inter-frame analysis unit 124 may execute the process of determining a frame group and the process of calculating an inter-frame projective transformation matrix independently or sequentially, or may execute both processes synchronously. Next, an example of the process of calculating an inter-frame projective transformation matrix will be described. FIG. 19 is a flowchart illustrating the process of calculating an inter-frame projective transformation matrix. FIG. 19 illustrates a case where the step of determining a frame group and the step of calculating an inter-frame projective transformation matrix are synchronized.

[0056] (Step S202) The inter-frame analysis unit 124 initially sets, as a group of consecutive frames, a predetermined number of frames that are adjacent in time and are fully expected to have an image overlap rate in the common area equal to or greater than a predetermined rate from the images of each frame shown in the image data. The inter-frame analysis unit 124 provisionally calculates an inter-frame projective transformation matrix between adjacent frames using Method 2 for each frame in the group of consecutive frames. The inter-frame analysis unit 124 then calculates the optical flow from the image of the first frame to the image of the last frame of the group of consecutive frames, and calculates the inter-frame projective transformation matrix H from the first frame to the last frame. f-long The inter-frame analysis unit 124 calculates the calculated inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the degree of conformance 1 between the inter-frame projective transformation matrices of adjacent frames in the continuous frame group, and calculates the cumulative value H f-multi The inter-frame analysis unit 124 calculates the calculated cumulative value H f-multi and calculates the degree of conformance 2 between the optical flow. The inter-frame analysis unit 124 compares degree of conformance 1 and degree of conformance 2, and if degree of conformance 2 is lower than degree of conformance 1, performs the processing of method 5 on the group of consecutive frames to adjust the inter-frame projective transformation matrix between adjacent frames. Thereafter, the inter-frame analysis unit 124 performs the processing of method 6 to further smooth the inter-frame projective transformation matrix between adjacent frames across multiple frames.

[0057] (Step S204) The inter-frame analysis unit 124 sets the frame immediately following the group of consecutive frames as the next frame and provisionally calculates an inter-frame projective transformation matrix from the last frame of the group of consecutive frames according to Method 2. (Step S206) The inter-frame analysis unit 124 calculates the optical flow from the image of the first frame of the group of consecutive frames to the image of the next frame, and calculates the inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the calculated inter-frame projective transformation matrix H f-long The degree of conformance 1 between the image and the optical flow is calculated.

[0058] (Step S208) The inter-frame analysis unit 124 calculates the total product of the inter-frame projective transformation matrices between adjacent frames in the group of consecutive frames and the inter-frame projective transformation matrices from the last frame to the next frame of the group of consecutive frames as the cumulative value H of the inter-frame projective transformation matrices from the start frame to the next frame. f-multi The inter-frame analysis unit 124 calculates the calculated cumulative value H f-multi and calculates the goodness of fit 2 between the optical flow. The inter-frame analysis unit 124 compares goodness of fit 1 and goodness of fit 2, and if goodness of fit 2 is lower than goodness of fit 1, it performs the process of method 5 on the final frame and the next frame of the consecutive frame group to adjust their inter-frame projective transformation matrices. At this stage, it is not necessary to consider the movements from the first frame of the consecutive frame group to the frame immediately before the final frame. The inter-frame analysis unit 124 then recalculates goodness of fit 2.

[0059] (Step S210) The inter-frame analysis unit 124 defines an area in which the start frame and the next frame in the consecutive frame group represent a common image as a common area, and calculates the ratio of the defined common area to the entire image as the image overlap rate. The inter-frame analysis unit 124 determines whether the calculated image overlap rate is equal to or less than a predetermined level. If the image overlap rate is equal to or less than the predetermined level (YES in step S210), the inter-frame analysis unit 124 proceeds to processing in step S216. In this case, the inter-frame analysis unit 124 determines that the next frame cannot be included in the immediately preceding consecutive frame group. This confirms one consecutive frame group. If the image overlap rate exceeds the predetermined level (NO in step S210), the inter-frame analysis unit 124 proceeds to processing in step S212.

[0060] (Step S212) The inter-frame analysis unit 124 determines whether the matching score 2 is equal to or lower than a predetermined level. If the matching score 2 is equal to or lower than the predetermined level (YES in step S212), the process proceeds to step S216. In this case, the inter-frame analysis unit 124 also determines that the next frame cannot be included in the immediately preceding group of consecutive frames. If the matching score 2 is higher than the predetermined level (NO in step S212), the process proceeds to step S214. (Step S214) The inter-frame analysis unit 124 adds the next frame to the group of consecutive frames as the final frame. Thereafter, the process proceeds to step S204.

[0061] (Step S216) The inter-frame analysis unit 124 searches for an angle-of-view overlap frame whose angle of view overlaps with the angle of view of the start frame of the determined consecutive frame group from past frames preceding the consecutive frame group. An angle-of-view overlap frame is a frame related to an image representing a common field of view, i.e., a frame for which the area ratio of the common area with the start frame is equal to or greater than a certain level and for which an inter-frame projection transformation matrix with a matching rate with the optical flow from the start frame is equal to or greater than a certain level is obtained. If an angle-of-view overlap frame is found (YES in step S216), the process proceeds to step S218. If an angle-of-view overlap frame is not found (NO in step S216), the process of FIG. 19 ends. (Step S218) The inter-frame analysis unit 124 identifies the angle-of-view overlap frame and performs the process of method 7 on an intermittent frame group including a frame in the latest consecutive frame group to reduce (minimize) the error in the inter-frame projection matrix belonging to the intermittent frame group. Furthermore, the inter-frame analysis unit 124 performs the processing of any one of methods 4 to 6 on consecutive adjacent frames existing between each frame included in the intermittent frame group, and adjusts these inter-frame projective transformation matrices so that they conform to the inter-frame projective transformation matrices between each frame included in the intermittent frame group. Thereafter, the processing of FIG. 19 ends.

[0062] Appendix 1. Use of a projective transformation matrix with limited degrees of freedom Appendix 1-1. About projective transformation matrices Coordinate transformation between coordinates representing the position of a point in one coordinate system and coordinates representing it in another coordinate system, or movement from one point to another within one coordinate system, or the correspondence between these two points, is mathematically expressed as a projective transformation. The matrix that represents this projective transformation is called a projective transformation matrix. For example, the relationship between point m on a two-dimensional coordinate system and point m' after projective transformation is expressed using equation (1) using the projective transformation matrix H.

[0063]

[0064] In equation (1), the coordinates of points m and m' are expressed in the homogeneous coordinate system by equations (2) and (3). The projection transformation matrix H is a 3-row, 3-column matrix expressed by equation (4). λ is a scale coefficient. λ can take any scalar value. Due to this arbitrariness, any element, for example, h22 The projective transformation matrix H may be normalized by setting the value of ∑ ...

[0065]

[0066]

[0067]

[0068] Supplementary Note 1-2. Restrictions on the degrees of freedom of the projective transformation matrix In camera geometry, the transformation from point M on the three-dimensional real space coordinate system to point m on the camera image coordinate system is expressed using the projective transformation matrix P as shown in equation (5).

[0069]

[0070] In equation (5), the coordinates of points M and m are expressed by equations (6) and (7), respectively. The projective transformation matrix P is a 3-row, 4-column matrix expressed by equation (8).

[0071]

[0072]

[0073] The projective transformation matrix P is expressed as the product of element matrices A, F, R, [E -T], which indicate the contribution of element parameters, as shown in Equation (8). The element matrix A is determined by the skew distortion s, the aspect ratio ρ, and the coordinates (u0, v0) of the intersection of the virtual image plane of the camera image and the optical axis of the camera, as shown in Equation (9). Depending on the specifications of the camera 20, s = 0, ρ = 1, (u0, v0) may be approximated as the coordinates of the center of the image. Furthermore, the values ​​of s, ρ, u0, and v0 may be determined in advance using a known camera internal parameter estimation technique. The element matrix F is determined by the focal length f, as shown in Equation (10). The element matrix R is a rotation matrix with three degrees of freedom. The element matrix R is determined by the orientation of the camera in the real space coordinate system, as shown in Equation (11). The vector T is determined by the coordinates (t x , t y , yz ) The matrix E is a 3-row, 3-column identity matrix.

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Regarding the positions m0 and m1 on the camera image coordinate system projected in two different frames of point M on the real-time coordinate system, the relationship shown in equation (13) is derived from equations (5) and (8). In equation (13), 0 and 1 indicate either of the two frames. The element matrices F and R and the vector T are variable depending on the frame.

[0080]

[0081] If the real space coordinate system is set to match the position and orientation of the camera in frame 0, and the scale of the real space coordinate system is set so that the focal length f0 in frame 0 is f0=1, then T0=[0,0,0] T , R0 = E, F0 = E. Furthermore, assuming that the camera position is constant in the real space coordinate system during shooting, i.e., T0 = T1, equation (13) can be rearranged as equation (14). Therefore, the projective transformation matrix H from frame 0 to frame 1 is given by equation (15). Since the element matrix A is an internal parameter that can be acquired from the camera 20 by the image processing device 10 or set for the camera 20, the degrees of freedom of the projective transformation matrix H are four, consisting of one degree of freedom for the element matrix F1 and three degrees of freedom for the rotation matrix R1. Furthermore, if no zoom operation is performed during shooting, the element matrix F1 becomes equal to the element matrix F0 (can be considered as F1 = E), and the degrees of freedom of the projective transformation matrix H are further reduced to three. In this projective transformation matrix H with limited degrees of freedom, components due to variable parameters with unlimited degrees of freedom change, while components due to fixed parameters with limited degrees of freedom do not change.

[0082]

[0083]

[0084] Supplementary Note 1-3. Quantification of Inter-Camera Image Displacement Using a Projection Transformation Matrix with Limited Degrees of Freedom The projective transformation between frames used in this application can sometimes be expressed with fewer degrees of freedom than the original degrees of freedom, depending on the image capture conditions. By using a projective transformation matrix with limited degrees of freedom, it is possible to prevent the detection of unrealistic image displacements, and the projective transformation matrix can be stably estimated. When changes in camera position during capture can be ignored and changes in camera orientation and zoom are allowed, the inter-frame analysis unit 124 can use a projective transformation matrix with four degrees of freedom modeled as in Equation (16).

[0085]

[0086] If changes in camera position and zoom can be ignored during shooting and changes in camera orientation are allowed, the inter-frame analysis unit 124 can use a projective transformation matrix with three degrees of freedom modeled as in equation (17).

[0087]

[0088] When determining the inter-frame projective transformation matrix, the inter-frame analysis unit 124 calculates the image transition between target frames as an optical flow for each block. The inter-frame analysis unit 124 can then estimate the inter-frame projective transformation matrix based on the above model using a regression method such as the RANSAC algorithm so as to increase the degree of conformance with the calculated optical flow. Specifically, the inter-frame analysis unit 124 can calculate the inter-frame projective transformation matrix by executing the procedure illustrated in FIG. 20.

[0089] The inter-frame analysis unit 124 repeats the processing of steps S302 to S306 a predetermined number of times, and then proceeds to the processing of step S308. (Step S302) The inter-frame analysis unit 124 randomly selects a number of optical flows according to the degrees of freedom of the model. For example, when using models with four degrees of freedom and three degrees of freedom, the number of optical flows is two. (Step S304) The inter-frame analysis unit 124 applies the above model to the selected optical flows. That is, the inter-frame analysis unit 124 determines the parameters of the model so that the movement indicated by the selected optical flows is as close as possible to the movement given by the inter-frame projective transformation matrix calculated using the model.

[0090] (Step S306) The inter-frame analysis unit 124 determines whether the motion indicated by the other optical flows not selected matches the motion given by the inter-frame projective transformation matrix calculated using the above model under the determined parameters. The inter-frame analysis unit 124 can determine whether the motion matches based on, for example, whether the difference between the two types of motion is equal to or less than a predetermined reference value. The inter-frame analysis unit 124 counts the number of other optical flows whose motion matches as the degree of match. (Step S308) The inter-frame analysis unit 124 determines the inter-frame projective transformation matrix that gives the highest degree of match as the regression value. The processing of FIG. 20 then ends.

[0091] In addition, when the position of the camera 20 is not changed but the orientation and zoom are changed, and the subject is sufficiently far away from the camera 20, it can be assumed that the image transition between frames is composed of elements such as translation (2 degrees of freedom, parameters: Δu, Δv), rotation (1 degree of freedom, parameter α), and enlargement or reduction (1 degree of freedom, parameter f). Therefore, the projective transformation matrix can be approximated as shown in equation (18). In this case, the number of optical flows to be randomly extracted in the regression is two. Note that, under the assumption that no enlargement or reduction occurs, the parameter f may be fixed at 1.

[0092]

[0093] The derivation of parameters that fit the two optical flows can be analytically achieved using the closed-form solution described below instead of parameter optimization. This reduces the computational cost. T ) to coordinate m1 ([u1, v1] T ) is rearranged as shown in equations (19) to (22) using the projection transformation matrix H shown in equation (18).

[0094]

[0095]

[0096]

[0097]

[0098] The coordinates m0 and m1 before and after the transformation are obtained from each of the two randomly extracted optical flows. 00 , m 10 and the coordinates m before and after the transformation for the other optical flow 1. 01 , m 11 are expressed by equations (23) and (24). Substituting equations (23) and (24) into equation (22) to solve the simultaneous equations yields equation (25), which is further transformed into equation (26). According to equation (26), the inter-frame analysis unit 124 determines the scaling factor f, the rotation angle α, and the translation amount (Δu, Δv) from the two optical flows.

[0099]

[0100]

[0101]

[0102]

[0103] On the other hand, if the position of the camera 20 changes during shooting, the degrees of freedom required to express that movement increase by three, resulting in seven degrees of freedom, including one for the focal length and three for the camera orientation. These degrees of freedom are close to the eight degrees of freedom of the 3-row, 3-column projective transformation matrix H shown in equation (27), and can be directly applied to this projective transformation matrix H. For the application, it is necessary to extract optical flows at four points. Even in this case, the inter-frame analysis unit 124 can analytically calculate the parameters using a closed-form solution.

[0104]

[0105] Supplementary Note 2. Use of a Projection Transformation Matrix Representing Micro-Movement The projection transformation matrix representing micro-movement refers to the micro-transformation matrix ΔH described above. The micro-transformation matrix ΔH is derived by randomly varying at least some element parameters belonging to the camera parameters. In this application, we propose adding the micro-transformation matrix ΔH to the inter-frame projective transformation matrix H and checking the compatibility of the inter-frame projective transformation matrix H after adding the micro-transformation matrix ΔH with the optical flow to search for an inter-frame projective transformation matrix H with higher compatibility with the optical flow. In a search method using a regression technique such as RANSAC from the optical flow, the true inter-frame projective transformation matrix cannot be guaranteed unless data with strictly true values ​​can be sampled. However, by performing the following method to expand the range of inter-frame projective transformation matrix candidates searched, compatibility with the optical flow and, ultimately, the accuracy of the inter-frame projective transformation matrix H can be improved.

[0106] The inter-frame analysis unit 124 generates the infinitesimal transformation matrix ΔH using a model that uses camera parameters as explanatory variables. The inter-frame analysis unit 124 selects a model with degrees of freedom corresponding to the shooting conditions. The inter-frame analysis unit 124 can acquire data, such as operation information such as zooming, and sensor signals indicating the movement state or orientation, from the camera 20 and determine the shooting conditions based on the acquired data. For example, when the position of the camera 20 does not change but the zoom changes, the inter-frame analysis unit 124 generates the infinitesimal transformation matrix ΔH using equation (28). Equation (28) corresponds to a model similar to equation (16) and includes variable factors: an element matrix F1 (1 degree of freedom) and a rotation matrix R1 (3 degrees of freedom). Therefore, the infinitesimal transformation matrix ΔH has four degrees of freedom. Note that if the inter-frame analysis unit 124 determines that no zoom change will occur, it fixes the element matrix F1, which includes the focal length, to F1 = E, and allows only the rotation matrix R1 to be variable. In equation (28), when F1=E and R1=E, the infinitesimal transformation matrix ΔH becomes a unit matrix, which indicates that no movement of the image occurs.

[0107] The inter-frame analysis unit 124 generates minute random noise centered on a value that gives element matrices F1 = E, R1 = E for each variable element parameter, and generates a minute transformation matrix ΔH containing the element matrices F1 and R1 as factors based on the generated random noise. For example, the inter-frame analysis unit 124 can generate the minute random noise by normalizing each sample value constituting a pseudo-random number sequence generated using a known method so that the sample values ​​fall within a predetermined center value and range. The range is set to be larger than calculation errors (e.g., quantization error, pixel spacing) but sufficiently smaller than typical fluctuations during shooting. If the inter-frame analysis unit 124 determines that no zoom change will occur, it fixes F1 = E, i.e., fixes the focal length f to 1, and allows only the orientation to be variable.

[0108]

[0109] The inter-frame analysis unit 124 may generate the infinitesimal transformation matrix ΔH using the model shown in equation (29). The model shown in equation (29) is the same as the model shown in equation (18) and has four degrees of freedom. In equation (29), if Δu = 0, Δv = 0, α = 0, and f = 1, the infinitesimal transformation matrix ΔH becomes a unit matrix, indicating that no image movement occurs. Then, infinitesimal random noise with center values ​​of 0, 0, 0, and 1 is generated for each element parameter Δu, Δv, α, and f, respectively, and the infinitesimal transformation matrix ΔH is generated using equation (29) based on the generated random noise. Note that if the inter-frame analysis unit 124 determines that no zoom change will occur, it fixes the focal length f to 1 and varies only Δu, Δv, and α.

[0110]

[0111] In addition, when it is already known how to improve the inter-frame projective transformation matrix H (i.e., when its ideal value or target value has been set), the inter-frame analysis unit 124 calculates the amount of change H required for this improvement before performing the minute transformation on the inter-frame projective transformation matrix H. diff If a pair of two frames related to the inter-frame projective transformation matrix H are adjacent, the inter-frame analysis unit 124 may apply a change amount H diff and the multiplied value is determined as the corrected inter-frame projective transformation matrix H. When the pair of two frames relating to the inter-frame projective transformation matrix H is separated in time by two or more frames, the inter-frame analysis unit 124 calculates the amount of change H diff is divided into differential components dH for each pair of adjacent frames, and multiplied by the differential components dH for each pair of adjacent frames to correct the inter-frame projective transformation matrix H for each pair of adjacent frames. For example, the ideal value H of the inter-frame projective transformation matrix from the start frame i to the final frame i+l in a certain frame group is i→i+l f is known, the inter-frame analysis unit 124 calculates the current value H of the inter-frame projective transformation matrix at that time as shown in equation (30). i→i+l f' is multiplied by the ideal value to obtain the change H diff Calculate.

[0112]

[0113] Then, the inter-frame analysis unit 124 calculates the calculated change amount H diff is calculated as the difference component dH to be distributed to the inter-frame projective transformation matrix of each set of l-1 adjacent frames in the frame group. Here, when the inter-frame projective transformation matrix is ​​derived using a reference frame, the set of adjacent frames from the starting frame to the reference frame is also calculated as the difference component H diff For example, the inter-frame analysis unit 124 can calculate the difference component dH using the model of equation (31). The model of equation (31) is the same as equation (28), and has four degrees of freedom.

[0114]

[0115] The inter-frame analysis unit 124 calculates the difference H diff The inter-frame analysis unit 124 divides the vectors m0 and m1 shown in equation (32) into the difference component dH for each pair of adjacent frames as follows, for example: diff Multiplying this by m i The inter-frame analysis unit 124 calculates the total product Π′ of the difference component dH for the number of pairs of adjacent frames in the frame group as shown in equation (34). l Multiplying by dH gives the estimated movement amount m i The inter-frame analysis unit 124 determines the target movement amount m i ' and estimated movement amount m i A difference component dH for each pair of adjacent frames that minimizes the sum of squared differences of " is searched for. Note that f1 in equation (35) represents a focal length parameter that constitutes the element matrix F1.

[0116]

[0117]

[0118]

[0119]

[0120] Supplementary Note 3. Weighted Moving Average (Smoothing) of Projection Transformation Matrices for a Group of Consecutive Frames The weighted moving average is equivalent to calculating a weighted sum of the element values ​​for each element, assuming similarity in the inter-frame projective transformation matrices between consecutive adjacent frames. In the example of equation (36), the weight for the target frame is generally (1-2α), and the weights for the previous frame immediately before it and the next frame immediately after it are each set to α. The coefficient α indicates the degree of smoothing effect achieved by the moving average. The coefficient α is a real number greater than 0 and less than 0.5. However, since the previous frame is not taken into account for start frame 0 in the group of frames, the weight for start frame 0 is set to 1-α. Since the subsequent frame is not taken into account for final frame l, the weight for final frame l is set to 1-α.

[0121]

[0122] The inter-frame projective transformation matrix H smoothed by the weighted moving average obtained based on Equation (36) i ' does not necessarily represent a weighted average of a geometrically strict projective transformation. Therefore, when performing the moving average, the inter-frame analysis unit 124 uses the inter-frame projective transformation matrix H i The compatibility between ' and the optical flow between the relevant frames is evaluated, and if the compatibility improves, the inter-frame projective transformation matrix H obtained by moving average is used. i ' is adopted, and if the compatibility does not improve, the original inter-frame projective transformation matrix H i Specifically, the inter-frame analysis unit 124 uses the individual inter-frame projective transformation matrices H i The procedure shown in FIG. 21 is executed for

[0123] (Step S402) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H between frame i and its adjacent frame. i The moving average value H is calculated based on a series of inter-frame projective transformation matrices within a predetermined interval including i(Step S404) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i Moving average value H i (Step S406) The inter-frame analysis unit 124 evaluates the compatibility 1 between the inter-frame projective transformation matrix H i The degree of conformance 2 between the optical flow and the frame is evaluated.

[0124] (Step S408) The inter-frame analysis unit 124 compares the fitness 1 with the fitness 2, and determines whether the fitness 1 is lower than the fitness 2. If the fitness 1 is lower (YES in step S408), the original inter-frame projective transformation matrix H i 21 is terminated. If the fitness level 1 does not decrease (NO in step S408), the inter-frame analysis unit 124 proceeds to step S410. (Step S410) The inter-frame analysis unit 124 uses the moving average value H i ' is used to calculate the inter-frame projective transformation matrix H i (Step S412) The inter-frame analysis unit 124 determines whether the number of repetitions of steps S404 to S410 has reached a predetermined number of repetitions. If it is determined that the number has not been reached (YES in step S412), the process returns to step S404. If it is determined that the number has been reached (NO in step S412), the process of FIG. 21 ends.

[0125] The inter-frame analysis unit 124 does not necessarily need to perform the moving average for all inter-frames, but may perform the moving average for the inter-frame projective transformation matrices for some inter-frames (e.g., between any one frame or more and less than l frames in one frame group).

[0126] Supplementary Note 4. Projection transformation matrix between real space plane coordinate system and camera image coordinate system (real space projective transformation matrix) H g The coordinate transformation from point M on the real space coordinate system to point m on the camera image coordinate system is expressed by equations (37) to (40) using a 3-row, 4-column projective transformation matrix P.

[0127]

[0128]

[0129]

[0130]

[0131] Assuming that the position of point M exists on the z=0 plane in real space, the relationships shown in equations (37) to (40) can be rearranged as shown in equations (41) and (42).

[0132]

[0133]

[0134] In equation (42), by omitting the third column of the projection transformation matrix P and the z coordinate of point m, a 3-row, 3-column real space projection transformation matrix H is obtained so as to satisfy the relationships shown in equations (43) to (45). g is derived.

[0135]

[0136]

[0137]

[0138] p 23 When equation (45) is normalized by setting =1, the relationships shown in equations (46) and (47) are derived.

[0139]

[0140]

[0141] Here, for n points with known coordinates arranged on a real space plane, the image coordinate values ​​(u k , v k ) (k=0, ..., n-1) are obtained, equation (47) can be expressed as a simultaneous equation in equation (48). If the matrix of the first term on the left side of equation (48) is B as in equation (49) and the vector on the right side is C as in equation (50), equation (48) can be rearranged as in equation (51). Equation (51) is 23 = 1, and convert matrix B and vector C into real space projection matrix H g is calculated.T B) has an inverse matrix, its rank must be 8. From this, if there are four or more known coordinate points, the real space projection transformation matrix H g It is shown that it is possible to calculate

[0142]

[0143]

[0144]

[0145]

[0146] In addition, the real space projection transformation matrix H g may be calculated by applying a known method. For example, when six or more points with known real space coordinates, including points not on a plane, are identified in the standard frame, the projection transformation matrix P can be calculated from the relationship between the real space coordinate values ​​of these known real space coordinate points and the camera image coordinate values ​​of the image corresponding points. Based on the element values ​​of this P and equation (45), H g is calculated.

[0147] Then, as shown in equation (54) derived from equations (52) to (53), the real space projection transformation matrix H g For the inter-frame projective transformation matrix H from the reference frame to other frames, f By multiplying by , the real space projection transformation matrix H g Therefore, the real space projection transformation matrix H g and for every pair of frames, the inter-frame projective transformation matrix H f is obtained, the real space projection transformation matrix H g is obtained.

[0148]

[0149]

[0150]

[0151] Appendix 5. Derivation of the projective transformation matrix P between the real space coordinate system and the camera image coordinate system Appendix 5-1. Calculation of focal length and camera extrinsic parameters Real space projective transformation matrix H g is expressed as a product of element matrices A, F, R, etc. using the camera parameters as shown in equation (55). The relationship shown in equation (56) holds for the first and second columns on both the left and right sides of equation (55). Then, the elements of the first and second columns of the rotation matrix R are expressed as a column vector r1 T , r2 T Then, equation (56) is transformed into equation (57). Note that in this example, the parameters s and ρ constituting the element matrix A are approximated to 0 and 1, respectively, but these parameters s and ρ may be set to other predetermined values.

[0152]

[0153]

[0154]

[0155] From the properties of the rotation matrix R, the column vector r0 T , r1 T is r0 T r0=r1 T r1, r0 T The relationship r1 = 0 is satisfied. According to this relationship, equation (58) can be obtained from equation (57). By transferring the focal length f to the left side of equation (58), two equations (59) related to f are obtained.

[0156]

[0157]

[0158] The calibration unit 130 may calculate the focal length f using either one of the two equations in equation (59), but the f obtained for each equation may differ due to the influence of errors related to the processing up to this point. Therefore, the calibration unit 130 may calculate the average value of f obtained from both equations as the focal length f. For example, the calibration unit 130 may calculate the average value of f obtained from each of the two equations as shown in equation (60). 2 The square root of the average value of the above may be calculated as the focal length f.

[0159]

[0160] The calibration unit 130 can calculate the rotation matrix R and the element matrix T according to equations (61) to (67) using the obtained focal length f. The rotation matrix R indicates the orientation of the camera 20. The element matrix T indicates the position of the camera 20. The element matrix F is a matrix composed of the focal length f of the camera 20.

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168] Appendix 5-2. Provisional Calculation of Camera Position and Recalculation of Focal Length and Camera Orientation The focal length f and camera position calculated using the method described in Appendix 5-1 may contain errors and differ significantly from the true values. On the other hand, the error between the camera orientation and the direction of the camera optical axis vector is relatively small. Furthermore, camera position errors tend to occur in the direction of the optical axis vector, which may be highly correlated with focal length errors. Therefore, if the camera position is assumed to remain constant during shooting, the calibration unit 130 provisionally calculates a candidate camera position under the assumption that the camera position is on the optical axis (see Figure 17). Here, any coordinate on the optical axis is formulated as being at position T+kr^2, which is k times the displacement indicated by the third row vector r^2 of the rotation matrix R (corresponding to a vector parallel to the camera optical axis) from the camera position T. The distance D between the coordinate T0 of the candidate camera position and the camera optical axis is given by Equation (68).

[0169]

[0170] The calibration unit 130 calculates D or D for all frames in a frame group or for all frames across a frame group that share a common region. 2 The calibration unit 130 calculates the focal length f and the rotation matrix R for each frame using the calculated coordinates T0 of the camera position candidate. The formula for calculating the focal length f is derived as follows: 0x , T 0y , T 0z ] T ) is substituted into equation (69), which is then rearranged into equation (70). For the rotation matrix R, T Since the relationship R = E holds, Equation (70) is further modified in sequence to Equations (71) to (74). Note that in this example, the parameters s and ρ constituting element matrix A are approximated to 0 and 1, respectively, but the parameters s and ρ may be set to other predetermined values. Noting that Equation (74) is a symmetric matrix, six equations related to the focal length f are obtained as shown in Equation (75). By eliminating the coefficient λ while satisfying the relationships shown in each equation of Equation (75), Equation (76) is obtained.

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179] The calibration unit 130 may calculate the focal length f using any one of the five equations shown in equation (76), but the focal length f calculated for each equation may differ due to the influence of errors related to the processing up to this point. Therefore, the calibration unit 130 calculates the focal length f calculated from any two to four equations in equation (76), or all five equations. 2 Alternatively, the square root of the average value of f may be calculated as the focal length f. The calibration unit 130 can calculate the rotation matrix R according to equations (77) to (82) using the calculated focal length f.

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186] Appendix 5-3. Bundle adjustment In the process up to Appendix 5-2, the real space projection transformation matrix H g The camera position candidate T0, the focal length f for each frame, and the camera orientation R provisionally calculated based on the above are algebraically appropriate solutions, but are not necessarily likely parameter values ​​in terms of camera geometry. The calibration unit 130 uses the provisionally calculated values ​​as initial values ​​and searches for camera parameters that minimize the reprojection error for known real space coordinate points. This process is called bundle adjustment. The procedure for bundle adjustment will be explained below. A known real space coordinate point M on a plane in the real space coordinate system is i ' and the coordinates of the reprojected points m iThe relationship between M and M' is given by equations (83) and (84). However, it is assumed that four or more image corresponding points have already been determined in each frame. Note that there are cases where the aggregated image corresponding points are located outside the angle of view of the image in the frame, but these image corresponding points may be used in the bundle adjustment process, or only some of them may be used, or only image corresponding points determined within the angle of view may be used without using these. In equation (84), M i ' denotes the real space coordinates of the real space coordinate known point detected in frame i. Also, m i is the real space coordinate known point M i ' corresponds to the image correspondence m i The coordinates of are shown.

[0187]

[0188]

[0189]

[0190] The calibration unit 130 defines the error between the image corresponding point and the reprojected point as the reprojection error e, and defines the sum of the squared value of the reprojection error shown in Equation (86) for each real-space coordinate known point and each frame in the frame group as the error index, and searches for the coordinate T0 of the camera position, the focal length f, and the rotation matrix R that minimize the error index for some or all frames (minimization of the reprojection error). The calibration unit 130 can use the coordinate T0, focal length f, and rotation matrix R of the candidate camera position obtained in the process up to Supplementary Note 5-2 as initial values. Note that the calibration unit 130 may set the coordinate T0 of the camera position obtained for some frames (e.g., the reference frame) as a default value, and use the focal length and rotation matrix R temporarily calculated based on this coordinate T0 as initial values ​​to search for the focal length f and rotation matrix R that minimize the error index for the other frames.

[0191]

[0192] Supplementary Note 5-4. Derivation of the projective transformation matrix P between the real space coordinate system and the camera image coordinate system and the real space projective transformation matrix H g , the inter-frame projective transformation matrix H fThe calibration unit 130 calculates the projective transformation matrix P according to equation (87) using the coordinates T0 of the camera position, the focal length f, and the rotation matrix R optimized in the process of Supplementary Note 5-3. The calibration unit 130 also performs calculations based on equations (88) and (89) to match the optimized parameters and calculates the real space projective transformation matrix H for each frame i. g and update the interframe projection matrix H f may be updated.

[0193]

[0194]

[0195]

[0196] Supplementary Note 5-5. Variability of Assumptions In the examples of Supplementary Note 5-1 to Supplementary Note 5-4, the camera parameters constituting element matrix A are assumed to be s = 0, ρ = 1, and u0 and v0 are assumed to be default values. However, this is not limited to this. Some or all of s, ρ, u0, and v0 may be pre-measured values. Furthermore, in the bundle adjustment described in Supplementary Note 5-3, some or all of these parameters may be included in the optimization target as variable variables. If movement of the camera 20 is detected, the camera position T0, which is assumed not to move, may be varied for each frame, and the step described in Supplementary Note 5-2 may be omitted. Furthermore, if no zoom operation is performed, the focal length f, which is assumed to vary for each frame, may be assumed to be constant between frames. The calculation processing unit 120 may determine the assumptions to be applied based on operation information acquired from the camera 20, various data indicating the movement state or orientation, etc., or an operation signal indicating an operation on the camera 20 input from the operation input unit 160.

[0197] As described above, the image processing device 10 according to this embodiment calculates an inter-frame projective transformation matrix that indicates a transition of images between frames, identifies image corresponding points corresponding to four or more known real-space coordinate points, which are points whose real-space coordinates are known on a real-space plane, in any frame of a frame group consisting of multiple frames, aggregates the image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on the inter-frame projective transformation matrix, calculates a real-space projective transformation matrix for the reference frame based on the known real-space coordinate points and the aggregated image corresponding points, calculates real-space projective transformation matrices for the other frames based on the inter-frame projective transformation matrices between the reference frame and the images of the other frames and the real-space projective transformation matrix, identifies image corresponding points in the other frames based on the inter-frame projective transformation matrix, and adjusts at least a portion of the camera parameters for each frame so as to reduce the difference between the estimated image corresponding points corresponding to the known real-space coordinate points estimated based on the camera parameters that constitute the real-space projective transformation matrix and the image corresponding points corresponding to the known real-space coordinate points.

[0198] According to this configuration, a real-space projective transformation matrix indicating the relationship between image corresponding points corresponding to known real-space coordinate points in a reference frame, which is one of multiple frames, and an inter-frame projective transformation matrix indicating image transitions between frames are determined. The real-space projective transformation matrix for each frame is then determined based on the determined real-space projective transformation matrix and the inter-frame projective transformation matrix of the reference frame. At least some of the camera parameters (e.g., focal length) are then determined by adjusting the image corresponding points for each frame to reduce the difference between the image corresponding points and the image corresponding points estimated from the corresponding known real-space coordinate points. This allows camera calibration to be performed under fewer constraints. For example, camera calibration can be performed for each frame even if markers placed at predetermined known real-space coordinate points do not always appear in the captured images. Furthermore, during camera calibration, camera parameters indicating changes in the field of view, such as zooming and rotation, are estimated. This facilitates image analysis across a wider range of shooting ranges and under a variety of shooting conditions.

[0199] This embodiment may be realized as follows: The image processing device 10 may determine an optical flow indicating image movement with a part in the second frame corresponding to the part in the first frame as an origin and an end point thereof, and calculate an inter-frame projective transformation matrix for a pair of the first frame and the second frame based on the determined optical flow.

[0200] The image processing device 10 may search for at least some camera parameters that indicate the transformation from the origin to the destination point corresponding to each set of a predetermined number of origins, and may search for an estimated value of the destination from an inter-frame projective transformation matrix based on the camera parameters searched for the origin, and an inter-frame projective transformation matrix that improves the degree of fit based on the optical flow that matches the destination point to the origin.

[0201] The image processing device 10 may calculate a first inter-frame projective transformation matrix, which is an inter-frame projective transformation matrix for a pair of a first frame and a third frame, which is a frame that is separated from the first frame by a predetermined time or more, calculate a second inter-frame projective transformation matrix, which is an inter-frame projective transformation matrix for a pair of the third frame and a second frame, which is a frame adjacent to the first frame, and calculate an inter-frame projective transformation matrix for the pair of the first frame and the second frame using the first inter-frame projective transformation matrix and the second inter-frame projective transformation matrix.

[0202] The image processing device 10 may execute an inter-frame projective transformation matrix search process to search for inter-frame projective transformation matrices for at least some pairs of adjacent frames so that the inter-frame projective transformation matrix for a pair of the first and last frames in a frame group consisting of multiple frames matches the product of the inter-frame projective transformation matrices for each pair of adjacent frames included in the frame group.

[0203] The image processing device 10 may apply a minute projection matrix, which is a projection transformation matrix corresponding to a random minute fluctuation in at least some of the element parameters that are elements of the camera parameters, to the inter-frame projection transformation matrix for at least some of the sets of frames included in a frame group consisting of a plurality of frames, to update the inter-frame projection transformation matrix, and may perform an inter-frame projection transformation matrix search process using the updated inter-frame projection transformation matrix.

[0204] The image processing device 10 may calculate, for each pair of frames, a moving average value of the inter-frame projective transformation matrix, calculate a correction value of the inter-frame projective transformation matrix based on the difference between the inter-frame projective transformation matrix and the moving average value, evaluate a first compatibility level that is the compatibility between the inter-frame projective transformation based on the correction value of the inter-frame projective transformation matrix and the optical flow from the origin block to the corresponding block, evaluate a second compatibility level that is the compatibility between the inter-frame projective transformation and the optical flow, and when the first compatibility level is not lower than the second compatibility level, update the inter-frame projective transformation based on the correction value of the inter-frame projective transformation matrix as the inter-frame projective transformation matrix.

[0205] The image processing device 10 may define, as a frame group, a plurality of consecutive frames in which the ratio of a common area representing a common field of view is equal to or greater than a predetermined ratio.

[0206] Another aspect of this embodiment may be a program for causing a computer to function as the image processing device 10 .

[0207] Another aspect of this embodiment may be an image processing system S1 including a camera 20 that captures an image and an image processing device 10.

[0208] Another aspect of the present embodiment may be a method in the image processing device 10, which calculates an inter-frame projective transformation matrix that indicates a transition of images between frames, identifies image corresponding points that correspond to four or more points with known real-space coordinates, which are points with known real-space coordinates on a real-space plane, in any frame of a frame group consisting of a plurality of frames, aggregates and determines the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculates a real-space projective transformation matrix of the reference frame based on the known real-space coordinates and the aggregated image corresponding points, calculates a real-space projective transformation matrix of the reference frame based on the inter-frame projective transformation matrix between the images of the reference frame and the other frames and the real-space projective transformation matrix of the reference frame, determines image corresponding points in the other frames based on the inter-frame projective transformation matrix, and determines the camera parameters so as to reduce a difference between the image corresponding points for each frame and estimated values ​​of image corresponding points that are estimated based on the camera parameters from the known real-space coordinate points that correspond to the image corresponding points.

[0209] Note that a portion of the image processing device 10 in the above-described embodiment, such as the arithmetic processing unit 120, may be implemented by a computer. In this case, a program for implementing this control function may be non-temporarily recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" refers to a computer system built into the image processing device 10, including hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the program may be a program for implementing a portion of the above-described functions, or may be a program that can be implemented in combination with a program already recorded in the computer system. Furthermore, part or all of the image processing device 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image processing device 10 may be individually implemented as a processor, or part or all of the functional blocks may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0210] The embodiments of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope of the gist of the present invention.

[0211] According to the image processing device, image processing system, image processing method, and program thereof according to the present disclosure, camera calibration can be achieved under conditions with fewer restrictions.

[0212] S1...image processing system, 10...image processing device, 20...camera, 30...display, 120...arithmetic processing unit, 122...image acquisition unit, 124...inter-frame analysis unit, 126...known coordinate point identification unit, 128...real space inter-image analysis unit, 130...calibration unit, 132...output processing unit, 140...storage unit, 150...input / output unit, 160...operation input unit

Claims

1. An image processing device that calculates an inter-frame projective transformation matrix that indicates the transition of images between frames, identifies image corresponding points that correspond to four or more known real-space coordinate points, which are points whose real-space coordinates are known on a real-space plane, in any frame of a frame group consisting of multiple frames, collectively determines the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculates a real-space projective transformation matrix of the reference frame based on the known real-space coordinate points and the image corresponding points, calculates a real-space projective transformation matrix of the other frame based on the inter-frame projective transformation matrix between the image of the reference frame and the image of the other frame and the real-space projective transformation matrix, determines image corresponding points in the other frame based on the inter-frame projective transformation matrix, and adjusts at least a portion of the camera parameters for each frame so as to reduce the difference between the estimated image corresponding points corresponding to the known real-space coordinate points, which are estimated based on camera parameters that constitute the real-space projective transformation matrix, and the image corresponding points corresponding to the known real-space coordinate points.

2. The image processing device according to claim 1, wherein an optical flow is determined for each part of a first frame that is a starting point, the optical flow indicating the movement of an image that ends at a part of a second frame that corresponds to the starting point, and the inter-frame projection transformation matrix for the pair of the first frame and the second frame is calculated based on the optical flow.

3. The image processing device according to claim 1, wherein for each set of a predetermined number of origin points, at least some camera parameters indicating a transformation from the origin point to an end point corresponding to the origin point are searched for, and an inter-frame projective transformation matrix that improves the degree of fit based on the optical flow that matches the end point to the origin point is searched for, and the estimated value of the end point is estimated from an inter-frame projective transformation matrix based on the camera parameters searched for the origin point.

4. The image processing device according to claim 1, wherein the image processing device calculates a first inter-frame projective transformation matrix which is an inter-frame projective transformation matrix for a pair of a first frame and a third frame which is a frame separated from the first frame by a predetermined time or more, calculates a second inter-frame projective transformation matrix which is an inter-frame projective transformation matrix for a pair of the third frame and a second frame which is a frame adjacent to the first frame, and calculates the inter-frame projective transformation matrix for the pair of the first frame and the second frame using the first inter-frame projective transformation matrix and the second inter-frame projective transformation matrix.

5. The image processing device according to claim 1, wherein an inter-frame projective transformation matrix search process is executed to search for inter-frame projective transformation matrices for at least some sets of frames so that the inter-frame projective transformation matrices of the first and last frames of a frame group consisting of multiple frames match the product of the inter-frame projective transformation matrices for each set of frames included in the frame group.

6. The image processing device according to claim 5, wherein an infinitesimal projection matrix, which is a projection transformation matrix corresponding to a random infinitesimal variation in at least some of the element parameters that are elements of the camera parameters, is applied to the inter-frame projection transformation matrix for at least some of the frame pairs included in a frame group consisting of a plurality of frames, thereby updating the inter-frame projection transformation matrix, and the inter-frame projection transformation matrix search process is executed using the updated inter-frame projection transformation matrix.

7. The image processing device according to claim 5, wherein the image processing device calculates a moving average value of the inter-frame projective transformation matrix for each set of frames, evaluates a first degree of compatibility which is the degree of compatibility between the optical flow and the image movement due to inter-frame projective transformation based on the inter-frame projective transformation matrix, evaluates a second degree of compatibility which is the degree of compatibility between the optical flow and the image movement due to inter-frame projective transformation based on the moving average value, and updates the inter-frame projective transformation matrix to the moving average value when the second degree of compatibility is not lower than the first degree of compatibility.

8. The image processing device according to claim 1, wherein a plurality of consecutive frames in which the ratio of a common area representing a common field of view is equal to or greater than a predetermined ratio are defined as the frame group.

9. A program for causing a computer to function as the image processing device according to claim 1.

10. An image processing system comprising: a camera that captures the image; and the image processing device according to claim 1.

11. An image processing method in an image processing device, comprising: calculating an inter-frame projective transformation matrix that indicates a transition of images between frames; identifying image corresponding points that correspond to four or more points with known real-space coordinates, which are points with known real-space coordinates on a real-space plane, in any frame of a frame group consisting of multiple frames; aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix; calculating a real-space projective transformation matrix for the reference frame based on the known real-space coordinate points and the image corresponding points; calculating a real-space projective transformation matrix for the other frame based on the inter-frame projective transformation matrix between the image of the reference frame and the image of the other frame and the real-space projective transformation matrix; determining image corresponding points in the other frame based on the inter-frame projective transformation matrix; and adjusting at least a part of the camera parameters for each frame so as to reduce the difference between the estimated values ​​of the image corresponding points corresponding to the known real-space coordinate points, which are estimated based on camera parameters that constitute the real-space projective transformation matrix, and the image corresponding points corresponding to the known real-space coordinates.

Images