Image processing device, image processing system, image processing method and program thereof

Abstract

To realize camera calibration under fewer constraints. [Solution] An inter-frame projective transformation matrix is ​​calculated, image corresponding points corresponding to four or more known real space coordinate points on a real space plane are identified in any frame of a frame group consisting of multiple frames, the image corresponding points are aggregated in a reference frame based on the inter-frame projective transformation matrix, a real space projective transformation matrix for the reference frame is calculated based on the known real space coordinate points and the image corresponding points, a real space projective transformation matrix for the other frame is calculated based on the inter-frame projective transformation matrix between the reference frame and images of the other frame, image corresponding points in the other frame are determined based on the inter-frame projective transformation matrix, and at least a portion of the camera parameters are adjusted so that the difference between the estimated values ​​of the image corresponding points corresponding to the known real space coordinate points estimated from the camera parameters that constitute the real space projective transformation matrix for each frame and the image corresponding points corresponding to the known real space coordinate points is reduced.

Description

[Technical field]

[0001] The present application relates to an image processing device, an image processing system, an image processing method, and a program thereof, and in particular to a camera calibration of a captured image. [Background technology]

[0002] Camera calibration is a process of acquiring parameters related to photography that are necessary for analyzing a captured image. The parameters related to photography are also called camera parameters. There are internal and external parameters for the camera parameters. The internal parameters include, for example, the focal length of the lens. The external parameters include the position and orientation of the camera. Camera calibration can also be considered as a process of identifying a homography matrix that is configured based on the camera parameters. The homography matrix represents the relationship between the coordinates of an object in a real space coordinate system and the coordinates of the object that appear in a camera image coordinate system. The real space coordinate system is a coordinate system that gives two-dimensional coordinates or three-dimensional coordinates that indicate an arbitrary position in a real space. The camera image coordinate system is a coordinate system that gives two-dimensional coordinates that indicate an arbitrary position in a captured image captured by a camera. In general, the success or failure and accuracy of video analysis that is configured from captured images are affected by camera calibration. Therefore, performing camera calibration accurately is very important in video analysis.

[0003] Conventionally, in camera calibration, the position, orientation, and zoom of the camera are fixed, an image for calibration is captured in advance without changing the angle of view during shooting, and a projection transformation matrix is ​​calculated using the image for calibration. Then, a separate image captured thereafter is analyzed using the calculated projection transformation matrix. When capturing an image for calibration, a point with a known real space coordinate (referred to as a "known real space coordinate point" in this application) is included in the field of view of the camera. A projection transformation matrix is ​​calculated using the relationship between the camera image coordinates of the known real space coordinate point that appears in the captured image and the real space coordinates in the real space.

[0004] The number of real space coordinate known points used to calculate the projective transformation matrix varies depending on the camera calibration method. For example, in the Direct Linear Transformation (DLT) method described in Non-Patent Document 1, the minimum number of known real space coordinate points required is six. Patent Document 1 proposes a method of using three known real space coordinate points by searching for camera internal parameter values ​​using an optimization method. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-116280 A [Non-patent literature]

[0006] [Non-Patent Document 1] Y. I Abdel-Aziz, HM Karara, Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. ASP symposium on close-range photogrammetry, Falls Church, VA, 1-19, 1971. [Non-Patent Document 2] Junji Takamatsu, Michiyoshi Ae, Norihisa Fujii, Development of the panning DLT method for a large measurement range, Journal of Physical Education, 42: 19-29, 1997 Summary of the Invention [Problem to be solved by the invention]

[0007] If the angle of view of the camera is fixed, the range of real space that appears in the captured image is limited. Therefore, a method has been proposed for analyzing an image captured while changing the angle of view. For example, in the panning DLT method described in Non-Patent Document 2, the object for which the angle of view is variable is limited to pan and tilt, and an image is captured while fixing the position of the camera using a tripod. Then, in an image captured so that a reference point whose real space coordinate does not change in the real space is always included in the field of view, a corresponding point is identified for each frame, and camera calibration is realized using the camera image coordinates of the identified point. However, the method of Non-Patent Document 2 requires that the reference point appears in the captured image in each frame. In addition, this method cannot handle changes in the field of view that accompany changes in the angle of view due to rotation, zoom, etc.

[0008] The present application has been made in consideration of the above points, and has as its objective the provision of an image processing device, an image processing system, an image processing method, and a program thereof that are capable of realizing camera calibration under fewer constraints. [Means for solving the problem]

[0009] (1) The present application has been made to solve the above-mentioned problems, and one aspect of the present application includes calculating an inter-frame projective transformation matrix that indicates a transition of images between frames, identifying image corresponding points that correspond to four or more known real space coordinate points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames, aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculating a real space projective transformation matrix for the reference frame based on the image corresponding points aggregated with the known real space coordinate points, calculating a real space projective transformation matrix for the other frame based on the inter-frame projective transformation matrix between the reference frame and images of the other frame and the real space projective transformation matrix, determining image corresponding points in the other frame based on the inter-frame projective transformation matrix, and adjusting at least a portion of the camera parameters for each frame so as to reduce a difference between an estimated value of the image corresponding point corresponding to the known real space coordinate point, which is estimated based on camera parameters that constitute the real space projective transformation matrix, and an image corresponding point corresponding to the known real space coordinate point.

[0010] (2) Another aspect of the present application is a method in an image processing device, comprising the steps of: calculating an inter-frame projective transformation matrix that indicates a transition of images between frames; identifying image corresponding points that correspond to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames; aggregating and determining the image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix; calculating a real space projective transformation matrix of the reference frame based on the image corresponding points aggregated with the real space coordinate known points; calculating a real space projective transformation matrix of the other frame based on the inter-frame projective transformation matrix between the reference frame and images of the other frame and the real space projective transformation matrix; determining image corresponding points in the other frame based on the inter-frame projective transformation matrix; and adjusting at least a portion of the camera parameters for each frame so as to reduce a difference between an estimated value of the image corresponding point corresponding to the known real space coordinate point, which is estimated based on camera parameters that constitute the real space projective transformation matrix, and the image corresponding point corresponding to the known real space coordinate point. Effect of the Invention

[0011] According to the present application, camera calibration can be realized under conditions with fewer restrictions, which facilitates image analysis in a wider range of shooting conditions and under various shooting conditions. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an image processing system according to an embodiment of the present invention. [Diagram 2] 1 is a schematic block diagram showing an example of the functional configuration of an image processing system according to an embodiment of the present invention; [Diagram 3] FIG. 1 is an explanatory diagram illustrating a projective transformation; [Figure 4] FIG. 1 is an explanatory diagram illustrating inter-frame projective transformation. [Diagram 5] FIG. 1 is a diagram illustrating an example of optical flow. [Figure 6]FIG. 1 is a first explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 7] FIG. 13 is a second explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 8] FIG. 13 is a third explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 9] FIG. 4 is a fourth explanatory diagram illustrating a method of calculating an inter-frame projective transformation matrix. [Figure 10] FIG. 5 is a fifth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 11] FIG. 6 is a diagram illustrating an example of a method for calculating an inter-frame projective transformation matrix; [Figure 12] FIG. 7 is a seventh explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 13] FIG. 8 is an eighth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 14] FIG. 9 is a ninth explanatory diagram illustrating a method for calculating an inter-frame projective transformation matrix. [Figure 15] 1 is an explanatory diagram illustrating an example of a relationship between inter-frame projective transformation and real space inter-image projective transformation; [Figure 16] FIG. 1 is an explanatory diagram illustrating bundle adjustment. [Figure 17] 10 is an explanatory diagram illustrating an example of a relationship between a camera position and a focal length error; [Figure 18] 1 is a flowchart illustrating an image processing method according to the present embodiment. [Figure 19] 13 is a flowchart showing a first example of a process for calculating an inter-frame projective transformation matrix. [Figure 20] 13 is a flowchart showing a second example of the process of calculating an inter-frame projective transformation matrix. [Figure 21] 11 is a flowchart illustrating a smoothing process of an inter-frame projective transformation matrix. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present application will be described with reference to the drawings. 1 is an explanatory diagram showing an overview of an image processing system S1 according to this embodiment. The image processing system S1 includes an image processing device 10 and a camera 20. The image processing device 10 acquires image data indicating images (sometimes referred to as "photographed images" in this application) captured at different times for each frame from the camera 20. The image processing device 10 performs calibration on the image indicated in the acquired image data, and estimates a homography matrix based on the parameters of the camera 20 (sometimes referred to as "camera parameters" in this application) for each frame. The homography matrix indicates the correspondence between a coordinate system (sometimes referred to as "real space coordinate system" in this application) set in a three-dimensional real space to be photographed and a two-dimensional coordinate system (sometimes referred to as "camera image coordinate system" or "image coordinate system" in this application) set in the photographed image. In the example of FIG. 3, a vector m indicating coordinates (u, v) in the photographed image ip corresponds to the product of a vector M indicating coordinates (X, Y, Z) in the real space, a homography matrix P, and a scale factor k. The homography matrix P is uniquely determined based on the camera parameters. The image processing device 10 analyzes the image for each frame based on the estimated homography matrix.

[0014] In the calibration according to the present embodiment, the image processing device 10 calculates an inter-frame projective transformation matrix indicating the transition of images between frames. The image processing device 10 specifies four or more image corresponding points corresponding to points whose real space coordinates are known on a real space plane in any frame of a frame group consisting of images of a plurality of frames. The image processing device 10 determines any one reference frame to be a reference among the frame group, and determines image corresponding points in the reference frame corresponding to image corresponding points in the arbitrary frame based on an inter-frame projective transformation matrix between the arbitrary frame in which the image corresponding points are specified and the reference frame. The image processing device 10 calculates a real space projective transformation matrix based on the correspondence between the real space coordinate known points and the image corresponding points for the reference frame. The image processing device 10 calculates a real space projective transformation matrix of the other frame based on the inter-frame projective transformation matrix between the reference frame and the other frame and the real space projective transformation matrix of the reference frame. The image processing device 10 determines image corresponding points in the other frame corresponding to image corresponding points in the reference frame based on the inter-frame projective transformation matrix between the reference frame and the other frame. The image processing device 10 determines, for each frame, the difference between the estimated value of the image corresponding point estimated based on the camera parameters constituting the real space projective transformation matrix and the coordinate of the image corresponding point corresponding to the real-time coordinate known point as a reprojection error, and adjusts some or all of the camera parameters so as to minimize the reprojection error. The image processing device 10 uses the camera parameters determined for each frame to analyze the captured image captured in that frame.

[0015] The camera 20 captures images showing the scene of a subject located within its field of view at different times. The camera 20 is, for example, a digital video camera. A digital video camera has a function of continuously capturing moving images. In general, a moving image is composed of still images for each frame captured at different times, and is captured or displayed at a predetermined frame rate (for example, 30 to 300 fps). The camera 20 transmits image data indicating the captured images to the image processing device 10.

[0016] According to the image processing system S1, calibration of captured images is realized for each frame, allowing for changes in the orientation and zoom of the camera 20 over time. However, it is necessary to set four or more feature points as known real space coordinates on a plane that spreads over a relatively wide area in real space, and to set the respective coordinates in advance in the image processing device 10. As the feature points, artificial or natural objects having morphological characteristics that can be easily distinguished from other objects or patterns are used.

[0017] The image processing system S1 according to this embodiment is applied to images obtained by continuously photographing a moving subject from a distant position, for example, in a stadium or other event venue. In the example of FIG. 1, a scene of an athlete running on a track Tr of an athletics stadium is photographed, and a hurdle installation position mark Mk on the track Tr is used as a feature point used for a real space coordinate known point. The feature points do not necessarily need to be photographed in advance. Four or more feature points do not necessarily need to be reflected in all frame images, and as long as they are displayed in at least a part of the multiple frames to be processed at one time, only three or less feature points may appear in any one of the multiple frames, or there may be a frame in which no feature points appear at all in the multiple frames. In this way, marks used in events such as competitions, or patterns or objects that do not hinder the movement of the subject and are not continuously blocked by the subject or other objects can be applied as real space coordinate known points. The feature points may be used as a reference for the real space coordinate system. For example, a horizontal plane on which multiple feature points are located may be defined as a plane whose vertical (z) coordinate value is zero, or any one of these, or the center of gravity, may be defined as the origin of the real space coordinate system.

[0018] The position and orientation of the camera 20 do not need to be fixed using a tripod or other device. Either or both of the position and orientation of the camera 20 can be changed. The orientation of the camera 20 can be changed by panning, tilting, or rotating. For example, the image processing device 10 according to this embodiment can also process images taken by a photographer holding the camera 20 while moving and adjusting the zoom to follow the movement of the subject. Therefore, even in a large space with one side or diameter of the shooting range exceeding 100 m, it is possible to process images taken by zooming or other camera work as long as the space has a known real space coordinate point set.

[0019] Next, an example of the functional configuration of the image processing system S1 according to this embodiment will be described. 2 is a schematic block diagram showing an example of the functional configuration of an image processing system S1 according to this embodiment. In the example of Fig. 2, the image processing system S1 includes an image processing device 10, a camera 20, and a display 30. The image processing device 10 includes an arithmetic processing unit 120, a storage unit 140, an input / output unit 150, and an operation input unit 160. The image processing device 10 may be configured as a general-purpose information processing device such as a personal computer or a workstation, or may be configured as a part of a dedicated image production device or image editing device.

[0020] The arithmetic processing unit 120 performs processing and control for implementing various functions of the image processing device 10. The arithmetic processing unit 120 may be configured using a dedicated member, or may be configured as a general-purpose computer system including a processor and a storage medium. The processor may implement the functions of each functional unit in cooperation with the storage medium and other hardware by reading a predetermined program stored in advance in the storage medium and executing the read program. In this application, execution of a program refers to executing processing instructed by an instruction written in the program. The processor includes, for example, a CPU (Central Processing Unit). The processor may include a type of arithmetic processing device different from the CPU, for example, a GPU (Graphic Processing Unit). Each functional unit of the arithmetic processing unit 120 will be described later.

[0021] The storage unit 140 stores data used for various processes in the arithmetic processing unit 120 and data acquired or generated by the arithmetic processing unit 120. The storage unit 140 includes an auxiliary storage device such as a hard disk drive (HDD: Hard-disk Drive) or a solid-state storage device (SSD: Solid State Drive). The input / output unit 150 inputs and outputs various data to and from other devices via a wired or wireless connection. The input / output unit 150 outputs image data input from the camera 20 to the arithmetic processing unit 120. The input / output unit 150 outputs display data input from the arithmetic processing unit 120 to the display 30. The input / output unit 150 includes, for example, an input / output interface. The input / output unit 150 may be connected to other devices via a communication network so as to be able to transmit and receive data. The operation input unit 160 receives a user's operation and generates an operation signal in response to the received operation. The operation input unit 160 outputs the generated operation signal to the arithmetic processing unit 120. The operation input unit 160 may have a general-purpose input device such as a mouse or a touch sensor, or may have a dedicated input device such as a button, a lever, or a dial.

[0022] The display 30 displays a display screen indicated by the display data input from the image processing device 10. The display 30 may be any type of display monitor, such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display.

[0023] One or both of the camera 20 and the display 30 may be configured separately from the image processing device 10, as exemplified in Fig. 2, or may be configured integrally with the image processing device 10. Furthermore, the operation input unit 160 may be configured integrally with the image processing device 10, as exemplified in Fig. 2, or may be configured separately. The touch sensor and the display 30 constituting the operation input unit 160 may be configured separately, or may be configured integrally as a touch panel.

[0024] Next, a description will be given of an example of the functional configuration of the arithmetic processing unit 120. The arithmetic processing unit 120 includes an image acquisition unit 122, an inter-frame analysis unit 124, a known coordinate point identification unit 126, a real space inter-image analysis unit 128, a calibration unit 130, and an output processing unit 132.

[0025] The image acquisition unit 122 receives image data input from the camera 20 using the input / output unit 150. The image acquisition unit 122 sequentially stores the acquired image data in the storage unit 140. The storage unit 140 accumulates image data acquired at different times. The inter-frame analysis unit 124 calculates an inter-frame projective transformation matrix that indicates the transition of images between frames at different times. The inter-frame analysis unit 124 reads image data to be processed from the storage unit 140, and calculates an optical flow that indicates the movement from an image of the current frame to an image of another frame shown in the read image data. The optical flow is expressed by a motion vector that indicates the amount of movement and the direction of movement for each part of the image. The optical flow is calculated so that a pattern at a certain part (starting point) in the current frame is as close as possible to a pattern at a part (ending point) after movement in another frame.

[0026] As shown in FIG. 4, the inter-frame analysis unit 124 calculates the coordinates [u, v, 1] of a certain part in the current frame i. T (T denotes the transpose of a vector or matrix), the pattern at this coordinate is transformed into the inter-frame projection transformation matrix H i→i+1 f The corresponding coordinates after the movement in the other frame i+1, [u', v', 1], are given by multiplying T The interframe projection transformation matrix H i→i+1 f However, in Figure 4, h 00 ,…,h 21 ,1 is the interframe projection transformation matrix H i→i+1 f In addition, k is a scale factor expressing the arbitrariness of the constant multiple of the inter-frame projective transformation matrix. The inter-frame analysis unit 124 searches for the inter-frame projective transformation matrix by executing a procedure such as, for example, regressing the optical flow calculated between frames. The inter-frame analysis unit 124 can also calculate the inter-frame projective transformation matrix between other sets of frames. For example, the inter-frame analysis unit 124 sequentially calculates the inter-frame projective transformation matrix from the frame to the next frame for each frame. The inter-frame analysis unit 124 stores the inter-frame projective transformation matrix calculated for each set of frames in the storage unit 140.

[0027] When calculating the optical flow, the inter-frame analysis unit 124 can use a known method, such as block matching, feature point matching, etc. According to block matching, the optical flow is calculated for each block that constitutes a part of the current frame and is made up of a predetermined number of pixels. Fig. 5 shows an example of the optical flow Of calculated by block matching. In the example of Fig. 5, the optical flow to the end block is shown for each starting block in which a plurality of pixels are arranged and divided at regular intervals by a grid. According to block matching, a block is searched for as the end block, which has a pattern that matches the pattern of the starting block most closely and has a matching degree equal to or greater than a predetermined reference value.

[0028] In the image illustrated in FIG. 5, an accurate optical flow may not be obtained in a region where a non-planar surface appears or in the vicinity thereof. Even in a region where a planar surface appears, an accurate optical flow may not be calculated for a region where the brightness or color is constant (monotone) or a region showing a pattern that is spatially repeated at a constant period. These optical flows including errors are factors that prevent an accurate inter-frame projective transformation matrix from being estimated. In order to reduce these effects, the inter-frame analysis unit 124 may determine the inter-frame projective transformation matrix by performing regression of the optical flow using a robust method. The inter-frame analysis unit 124 may use, for example, the RANSAC (Random Sample Consensus) method. Specific examples of the calculation method of the inter-frame projective transformation matrix will be described later as methods 1 to 7.

[0029] Returning to FIG. 2, the coordinate-known point identification unit 126 identifies, for each of four or more real space coordinate-known points in the real space represented in the image, the portion in which the real space coordinate-known point appears as an image corresponding point (digitizes the real coordinate-known points). The coordinate known point identification unit 126 causes the display 30 to display a display screen on which at least one frame of an image to be analyzed (which may be referred to as a "target image" in the present application) is arranged for the output processing unit 132. The coordinate known point identification unit 126 specifies a part within the target image indicated by an operation signal input from the operation input unit 160 as an image corresponding point. Thereby, while viewing the display screen displayed on the display 30, the user can recognize the feature points where the coordinate known points are set and indicate the part where the feature points appear by an operation. In the image of the track Tr illustrated in FIG. 1, the position of the mark Mk is indicated as the image corresponding point. Regarding the real-space coordinate known points to be specified as the target of the image corresponding point, it is sufficient that four or more appear in any frame in the group of frames to be processed at one time. The four or more different real-space coordinate known points may all be represented in one frame, or may be distributed and represented in two or more frames. The coordinate known point identification unit 126 stores in the storage unit 140 the coordinate known point information indicating the coordinates of the frame and the image corresponding point specified for each real-space coordinate known point.

[0030] The real-space image-interval analysis unit 128 determines one frame of the group of frames to be processed as a reference frame. The real-space image-interval analysis unit 128 can determine, for example, the first frame, or the central frame, or the last frame in the group of frames to be processed, or the frame in which the coordinates of the most image corresponding points for the real-space coordinate known points indicated in the coordinate known point information stored in the storage unit 140 are specified, as the reference frame. The real-space image-interval analysis unit 128 determines the inter-frame projective transformation matrix from the frame in which the image corresponding point of the real-space coordinate known point is specified to the reference frame using the inter-frame projective transformation matrix stored in the storage unit 140. The real-space image-interval analysis unit 128 determines the inter-frame projective transformation matrix H from the frame j (j < i) preceding the reference frame i to the reference frame i as H j→i f as H i-1→i f …H j+1→j+2 f H j→j+1 fThe real space inter-image analysis unit 128 calculates an inter-frame projective transformation matrix H j→i f , (H i→i+1 f ) -1 …(H j-2→j-1 f ) -1 (H j-1→j f ) -1 The real space inter-image analysis unit 128 can determine the coordinates of the image corresponding points in the reference frame by multiplying the coordinates of the image corresponding points specified in the frame by the determined inter-frame projective transformation matrix. In this way, the image corresponding points are aggregated in the reference frame, and the coordinates of the image corresponding points in the reference frame are determined for each of the four or more real space coordinate known points.

[0031] The real space inter-image analysis unit 128 calculates a real space projection transformation matrix H for transforming an arbitrary point M′ on a plane in the real space coordinate system to a point m in the camera image coordinate system based on the real space coordinates and the image corresponding points for each real space coordinate known point in the reference frame k. k g Calculate the real space projection transformation matrix H g The calculation method will be described later in Appendix 4. The real space inter-image analysis unit 128 calculates a real space projective transformation matrix H k g The interframe projection transformation matrix H k→j f Multiplying by , we obtain the real space projection transformation matrix H j g FIG. 15 shows the real space projection transformation matrix H i-1 g Then, the inter-frame projective transformation matrix H i f The matrix H obtained by multiplying i f H i-1 g Let H be the real space projection transformation matrix for frame i. ig In addition, the real space projection transformation matrix H i g Then, the interframe projection transformation matrix H i+1 f By multiplying the matrix H i+1 g The real space inter-image analysis unit 128 stores the real space projective transformation matrix calculated for each frame in the storage unit 140. The real space projective transformation matrix will be described later in Supplementary Note 4.

[0032] The calibration unit 130 adjusts the camera parameters so that the image corresponding points on the camera image coordinate system corresponding to the real space coordinate known points for each frame and the estimated values ​​of the image corresponding points transformed from the real space coordinate known points in that frame using the camera parameters constituting the real space projective transformation matrix match as closely as possible (bundle adjustment). The camera parameters to be adjusted are limited to some element parameters (for example, focal length), and the other element parameters may be fixed to a constant value. The calibration unit 130 determines the image corresponding points in each of the other frames using an inter-frame projective transformation matrix for each image corresponding point in the reference frame. The calibration unit 130 sets the camera parameters related to the real space projective transformation matrix as initial values, and calculates the estimated values ​​of the image corresponding points estimated using the camera parameters for the real space coordinate known points. The calibration unit 130 sets the difference between the image corresponding points and their estimated values ​​as a reprojection error, and searches for camera parameters that minimize the reprojection error. 16 illustrates a case where the difference in coordinate values ​​between image corresponding points m and their estimated values ​​m' is taken as a reprojection error, and the camera parameters are adjusted so as to reduce the reprojection error. The calibration unit 130 stores the camera parameters determined for each frame in the storage unit 140. The camera parameters will be described later in Supplementary Note 5. In particular, the bundle adjustment will be described in Supplementary Note 5-3. The calibration unit 130 may analyze the image of the frame by using the camera parameters determined for each frame. The calibration unit 130 stores in the storage unit 140 corrected image data indicating the corrected image.

[0033] The output processing unit 132 outputs various types of data to the outside of the image processing device 10. The output processing unit 132 may configure a display screen and output display data indicating the configured display screen to the display 30 to display the display screen. The output processing unit 132 may analyze or correct the acquired image using the camera parameters obtained by the calibration unit 130. The output processing unit 132 may output display data representing the corrected image to the display 30, thereby displaying the corrected image. When the output processing unit 132 receives a corrected image request command from another device via the input / output unit 150, the output processing unit 132 may transmit corrected image data indicating the corrected image to the other device that is the source of the transmission. The output processing unit 132 may output display data representing the analysis results to the display 30, thereby displaying the analysis results. When the output processing unit 132 receives an analysis command from another device via the input / output unit 150, the output processing unit 132 may transmit analysis data indicating the analysis result to the other device that sent the analysis command or to another device.

[0034] Next, an example of an image processing method according to this embodiment will be described with reference to a flowchart shown in FIG 18. (Step S102) The image acquisition unit 122 of the image processing device 10 receives image data indicating a captured image from the camera 20, and accumulates the data in the storage unit 140 sequentially. (Step S104) The inter-frame analysis unit 124 calculates an inter-frame homography matrix that indicates the motion of images between different frames. (Step S106) The coordinate known point identification unit 126 displays the target image to be processed on the display 30, and identifies image corresponding points for each of the four or more real space coordinate known points based on coordinates indicated by an operation signal input from the operation input unit 160. (Step S108) The real space inter-image analysis unit 128 determines image corresponding points in the image in the reference frame using the image corresponding points for the real space coordinate known points specified in any frame and the inter-frame projective transformation matrix. The real space inter-image analysis unit 128 calculates a real space projective transformation matrix for the reference frame based on the correspondence between the real space coordinates of the real space coordinate known points and the coordinates of the image corresponding points in the reference frame. The real space inter-image analysis unit 128 calculates a real space projective transformation matrix for each frame using the inter-frame projective transformation matrix for each frame set and the real space projective transformation matrix for the reference frame.

[0035] (Step S110) The calibration unit 130 determines image corresponding points in each frame using an inter-frame projective transformation matrix for each image corresponding point in the reference frame. The calibration unit 130 sets the camera parameters related to the real space projective transformation matrix as initial values, and calculates estimates of image corresponding points estimated using the camera parameters for known real space coordinate points. The calibration unit 130 sets the difference in coordinate values ​​between the image corresponding points and their estimated values ​​as a reprojection error, and searches for some or all of the camera parameters so as to minimize the reprojection error (minimization of the reprojection error). (Step S112) Output processing unit 132 executes processing using the obtained camera parameters. For example, output processing unit 132 corrects the image of the frame using the obtained camera parameters. Output processing unit 132 outputs display data indicating a display screen including the corrected image to display 30, and displays the display screen. Output processing unit 132 transmits the obtained camera parameters to another device via input / output unit 150, for example. Thereafter, the processing of FIG. 18 ends.

[0036] Next, a specific example of a method for calculating the inter-frame projective transformation matrix according to this embodiment will be described. Method 1: The inter-frame analysis unit 124 may ignore the optical flow calculated for an area in which a moving object appears in one or both of an image of a starting frame (hereinafter sometimes referred to as a "starting frame", e.g., a current frame) and an image of an ending frame (hereinafter sometimes referred to as an "ending frame", e.g., a next frame), and may calculate an inter-frame projective transformation matrix using the optical flow of other areas. The inter-frame analysis unit 124 may calculate an inter-frame projective transformation matrix by applying a robust regression method to the optical flow of other areas.

[0037] In the example of FIG. 5, optical flows originating from an area in the current frame where a running athlete appears and an area within a certain width from the outer edge are ignored. The inter-frame analysis unit 124 can estimate an area in the target image where a moving object appears. As the known image processing technology, an image recognition technology for recognizing an image area of ​​a moving object such as a human may be used, or the following method may be used. For example, the inter-frame analysis unit 124 obtains the number distribution of optical flows for each pair of movement amount and movement direction for the image of the start frame, and determines the pair of movement amount and movement direction with the maximum number as the most frequent value of the optical flow. The inter-frame analysis unit 124 regards an optical flow whose deviation from the most frequent value is within a certain value as representing the global movement of the entire image from the start frame to the end frame, and estimates an area related to an optical flow whose deviation from the most frequent value exceeds a certain value as an area where a moving object appears. For example, when the degree of coincidence between a pattern shown in a peripheral area of ​​an optical flow start point in a start frame and a pattern shown in a peripheral area of ​​an optical flow end point in an end frame does not meet a predetermined reference value of the degree of coincidence, the inter-frame analysis unit 124 estimates the area related to the optical flow as an area where a moving object appears. According to these methods, the amount of processing can be reduced compared to image recognition technology.

[0038] Method 2: When calculating an interframe projective transformation matrix between a start frame and an end frame that is close in time (for example, between adjacent frames), the interframe analysis unit 124 may calculate the product of the interframe projective transformation matrix from the start frame to the reference frame and the interframe transformation matrix from the reference frame to the end frame as the interframe projective transformation matrix from the start frame to the end frame. The reference frame is a frame that is further away in time from the start frame and the end frame than the interval between them. In the example of FIG. 6, the interframe analysis unit 124 calculates the interframe projective transformation matrix H i→i+a f and the interframe projection transformation matrix H i+a→i+1 f Product of H i+a→i+1 f H i→i+a f Let H be the interframe projection transformation matrix from the current frame i to the next frame i+1. i→i+1 f The calculated interframe projection transformation matrix is ​​calculated as a matrix a. Since the deformation and movement of a moving object are relatively small between frames that are close in time, it may be difficult to estimate the area in which the moving object appears. If the optical flow related to the moving object cannot be excluded when estimating the interframe projection transformation matrix, an error occurs in the calculated interframe projection transformation matrix. The influence of the error can be reduced by deriving the interframe projection transformation matrix between frames that are close in time from the interframe projection transformation matrix between frames that are spaced apart from each other by a longer time interval, as in the present method. The reference frame a may be subsequent to the end frame or prior to the start frame. The reference frame a may be, for example, the start frame or the last frame of a group of frames (described later).

[0039] Method 3: The inter-frame analysis unit 124 may determine the inter-frame homography matrix by restricting the degree of freedom of the change in the camera parameters between frames. The inter-frame analysis unit 124 may also adjust the degree of freedom depending on the operation status of the camera 20. The inter-frame projective transformation matrix is ​​a 3-by-3 matrix with 8 degrees of freedom that is uniquely determined based on the camera parameters. In other words, the degrees of freedom of the inter-frame projective transformation matrix correspond to the degrees of freedom of the camera parameters. Restricting the degrees of freedom of the camera parameters means assuming that only some of the camera parameters change, and ignoring changes in the parameters that are the other elements. The camera parameters include position (3 degrees of freedom), orientation (3 degrees of freedom), and focal length (1 degree of freedom).

[0040] For example, if the movement distance of the camera 20 between frames is less than a predetermined reference value, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix based on a four-degree-of-freedom model in which the orientation and focal length are variable, under the assumption that the position is constant. The inter-frame analysis unit 124 can estimate the distance traveled by, for example, acquiring signals indicating the camera's motion state detected by position, speed, and acceleration sensors provided in the camera 20, and processing the acquired signals together with the time and time intervals of each frame.

[0041] For example, when the range of real space captured by camera 20 is sufficiently far from the position of camera 20 and it is determined that the change in position of camera 20 is relatively small compared to the range of real space captured in the image, inter-frame analysis unit 124 can determine an inter-frame projective transformation matrix based on a four-degree-of-freedom model in which the orientation and focal length are variable, under the assumption that the position is constant.

[0042] When no zoom operation is performed, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix under the assumption that the focal length is constant, with only the other parameters being variable. If the position of the camera 20 can be assumed to be constant and no zoom operation is performed, the inter-frame analysis unit 124 can determine the inter-frame projective transformation matrix based on a three-degree-of-freedom model in which the camera position and focal length are fixed and only the orientation is variable. The inter-frame analysis section 124 can determine whether or not a zoom operation is to be performed, for example, depending on whether or not an operation signal indicating a zoom command to the camera 20 is input. By restricting the degrees of freedom of the camera parameters as described above, the risk of calculating an unrealistic interframe projective transformation matrix from an optical flow that includes some errors is reduced, and the interframe projective transformation matrix is ​​calculated stably with a certain degree of accuracy. Note that the restrictions on the degrees of freedom of the camera parameters will be described later in Appendix 1.

[0043] The interframe projective transformation matrices between adjacent frames calculated individually as described above can also be used to calculate an interframe projective transformation matrix between frames separated in time by multiplying them together. However, the interframe projective transformation matrix between frames separated in time calculated in this manner accumulates errors contained in the interframe projective transformation matrices between the individual adjacent frames. The interframe analysis unit 124 can reduce the accumulated errors by applying any one of the processes related to the following methods 4 to 7, or any combination thereof.

[0044] Method 4: The inter-frame analysis unit 124 may search for an inter-frame projective transformation matrix for each pair of frames in a group of frames consisting of a plurality of frames, so that the concordance between the inter-frame projective transformation matrix between the frames obtained by multiplying the optical flow between any two frames (for example, the start frame to the last frame of the group of frames) and the inter-frame projective transformation matrix between the other frames is high. In the example of Fig. 7, the total product of the inter-frame projective transformation matrix from the start frame i to the last frame i+l of the group of frames and the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames is used as an estimate of the inter-frame projective transformation matrix from the start frame i to the last frame i+l, and the inter-frame projective transformation matrix for each pair of adjacent frames in the group of frames is searched for while considering the concordance between the optical flow between the two frames and the estimated value of the inter-frame projective transformation matrix from the start frame i to the last frame i+l.

[0045] The frame section to be evaluated for compatibility is not limited to a section starting from the start frame and ending at the final frame. This section may be a section divided by two different frames in the frame group. FIG. 8 illustrates a case in which the product of the total product of the inverse matrices of the interframe projective transformation matrix for each pair of adjacent frames from the start frame i to the penultimate frame i+l-1 and the interframe projective transformation matrix from the start frame i to the final frame l is used as an estimate value between the interframe projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l. The interframe analysis unit 124 searches for the interframe projective transformation matrix for each pair of adjacent frames from the start frame i to the penultimate frame i+l-1 in the frame group and the interframe projective transformation matrix from the start frame i to the final frame l, while taking into consideration the need to increase the compatibility between the estimated value of the interframe projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l and the optical flow between the two frames.

[0046] Assuming that the inter-frame projective transformation matrices for some of the frame pairs are default, the inter-frame analysis unit 124 may perform the above-mentioned inter-frame projective transformation matrix search for only the remaining frame pairs, and may leave the inter-frame projective transformation matrices for some of the frame pairs fixed to the default values ​​and exclude them from the inter-frame projective transformation matrix search. The example of Fig. 9, like the example of Fig. 7, is based on searching for an interframe projective transformation matrix for each pair of adjacent frames in the frame group that has a higher compatibility with the optical flow between the frames, using the total product of the interframe projective transformation matrices for each pair of adjacent frames in the frame group as an estimate of the interframe projective transformation matrix from the start frame i to the final frame i+l. Here, the interframe analysis unit 124 searches only for the interframe projective transformation matrix from frame i2 to frame i2+1, and regards the interframe projective transformation matrices for each pair of other frames as default values.

[0047] In the example of Fig. 10, similarly to the example of Fig. 8, the inter-frame analysis unit 124 searches for an inter-frame projective transformation matrix for each pair of adjacent frames from the start frame i to the second-to-last frame i+l-1 in the frame group and an inter-frame projective transformation matrix from the start frame i to the final frame l, so as to obtain a higher compatibility between the estimated inter-frame projective transformation matrix from the penultimate frame i+l-1 to the final frame i+l and the optical flow between the two frames. Here, the inter-frame analysis unit 124 searches only for an inter-frame projective transformation matrix from frame i2 to frame i2+1, and regards the inter-frame projective transformation matrices for each pair of frames other than the above as default values. As the inter-frame projective transformation matrix assumed to be the default, for example, a roughly accurate inter-frame projective transformation matrix, that is, an inter-frame projective transformation matrix obtained in advance with a degree of accuracy that is expected to be accurate, is applied.

[0048] In method 4, the longer the period from the start frame i to the final frame i+l, the greater the effect of reducing the accumulated error. However, in order to calculate an optical flow that can appropriately estimate the inter-frame projective transformation matrix between both frames, the ratio (hereinafter, sometimes referred to as the "image overlap rate") of the area in the image in which the area in the real space common to both frames appears (hereinafter, sometimes referred to as the "common area") must be at least a certain level. Therefore, the inter-frame analysis unit 124 identifies the common area after the fact based on the inter-frame projective transformation matrix searched for each set of different frames. The inter-frame analysis unit 124 may determine, as a series of frames, a section in which the identified common area occupies a certain ratio or more. The image overlap rate of the common area is set in advance based on a balance with the influence of the error contained in the optical flow. A series of frames consisting of a plurality of frames in which a common scene appears is determined based on the image overlap rate of the common area. The inter-frame analysis unit 124 may execute method 4 for each predetermined frame group, or may perform the search for the inter-frame projective transformation matrix and the setting of the frame group in parallel. A common frame may exist between a plurality of frame groups. In that case, overlapping of time series between the frame groups is permitted. The inter-frame analysis unit 124 can reduce accumulated errors across a plurality of frame groups by executing method 4 for each frame group.

[0049] Method 5: The inter-frame analysis unit 124 may perform a micro-transformation on the inter-frame projective transformation matrices of some or all of the frame pairs in the frame group, and check the compatibility of the transformed inter-frame projective transformation matrices with the optical flow related to method 4 to search for an inter-frame projective transformation matrix that has higher compatibility. The projective transformation matrix is ​​expressed as the product of element matrices indicating the components for each element parameter (see Supplementary Note 1). An element parameter refers to an individual parameter that is an element of a camera parameter. Examples of element parameters include the position, orientation, and focal length of the camera 20. In the infinitesimal transformation, the inter-frame analysis unit 124 assumes that some or all of the element parameters have fluctuated infinitesimally at random, and can calculate the product of element matrices corresponding to the amount of fluctuation related to the fluctuation as the infinitesimal transformation matrix ΔH.

[0050] Fig. 11 illustrates a case where an interframe projection transformation matrix that has a higher total compatibility between the interframe projection transformation matrix and the optical flow between adjacent frames and between the start frame and the final frame is searched for by multiplying each interframe projection transformation matrix between adjacent frames by an infinitesimal transformation matrix in the method illustrated in Fig. 7. Here, the interframe analysis unit 124 compares the compatibility 1 obtained by using the interframe projection transformation matrix before transformation with the compatibility 2 obtained by using the interframe projection transformation matrix after transformation, and if the compatibility 2 is higher than the compatibility 1, updates the interframe projection transformation matrix to the interframe projection transformation matrix after transformation as a new interframe projection transformation matrix, and does not update the interframe projection transformation matrix otherwise. Then, until the compatibility does not improve or a predetermined number of repetitions is reached, the interframe analysis unit 124 repeats the process of infinitesimal transformation of the interframe projection transformation matrix, comparison of the compatibility before and after transformation, and update of the interframe projection transformation matrix. When roughly accurate interframe projective transformation matrices are obtained between all adjacent frames in a frame group, it may be difficult to further reduce the accumulated error that appears in the interframe projective transformation matrices between temporally distant frames even by repeating a robust regression method such as the RANSAC method. Even in such cases, it is possible to further search for interframe projective transformation matrices that are not candidates using the above method by adding random small transformations. This provides an opportunity to reduce the accumulated error.

[0051] If an ideal value (or target value) of the interframe projective transformation matrix between two frames in a certain frame group is set or known, the interframe analysis unit 124 may make a small change to the interframe projective transformation matrix between other frames based on the ideal value to check whether the degree of conformity with the optical flow is improved. Here, the interframe analysis unit 124 calculates the difference H diff and distributing it to each pair of frames to calculate the difference component dH. The inter-frame analysis unit 124 corrects the inter-frame projective transformation matrix by applying the difference component dH distributed to the pair of frames (see FIG. 12). The inter-frame analysis unit 124 then determines whether the degree of conformance with the optical flow related to the frame group is improved by the corrected inter-frame projective transformation matrix compared to the degree of conformance based on the inter-frame projective transformation matrix before the correction. If the inter-frame analysis unit 124 determines that the degree of conformance is improved, it updates the inter-frame projective transformation matrix to the corrected one, and if it determines that the degree of conformance is not improved, it does not update the inter-frame projective transformation matrix to the corrected one. Note that the minute transformation will be described later in Supplementary Note 2.

[0052] Method 6: The inter-frame analysis unit 124 may calculate a weighted average of inter-frame projective transformation matrices between multiple frames that are consecutive in time series with respect to the target frames for each adjacent frame to be processed. The inter-frame analysis unit 124 calculates the compatibility between the inter-frame projective transformation matrix and the optical flow before and after the weighted average calculation. If the sum of the calculated compatibility between the frames does not decrease due to the weighted average, the inter-frame analysis unit 124 updates the weighted average as a new inter-frame projective transformation matrix. If the sum of the calculated compatibility decreases due to the weighted average, the inter-frame analysis unit 124 adopts the inter-frame projective transformation matrix before the weighted average calculation. This allows the inter-frame analysis unit 124 to accurately estimate the movement of the image while reducing the fluctuation of the projective transformation matrix between frames. The weighted average will be described later in Supplementary Note 3.

[0053] Method 7: The inter-frame analysis unit 124 may determine whether a common region that represents a common real space is shared even between frames that are not consecutive in time, and may execute any of methods 4 to 6 between frames that share a common region. FIG. 13 shows a case where a group of frames that share a common region (shown as three rectangular frames on the left and one rectangular frame on the right in FIG. 13) are generated at intervals in time, and the inter-frame projective transformation matrices of intermittent frames that intermittently connect these frames are calculated in chronological order as H0 f ,… , H4 f The inter-frame analysis unit 124 calculates the total product H intermit f with the total product of the inter-frame projective transformation matrices between all adjacent frames in between, and execute any one of the processes of methods 4 to 6. After performing a process of calculating or updating the inter-frame projective transformation matrix between frames included in the intermittent frame group, the inter-frame analysis unit 124 may set the inter-frame projective transformation matrix obtained by the process as a default value and execute any one of the processes of methods 4 to 6 for all adjacent frames included in each of the frames (see FIG. 14 ).

[0054] Note that, before performing processing on the intermittent frame groups as illustrated in Figures 13 and 14, the inter-frame analysis unit 124 may perform any of methods 4 to 6 on each consecutive frame group that includes those frames. When determining that a frame is one that shares a common area, the inter-frame analysis unit 124 can determine, for example, that a frame included in a frame group has a common area shared with another frame where the ratio of the portion of the common area shared with that frame group is equal to or greater than a predetermined ratio, as a frame that shares a common area with that frame group. When capturing an image, the user may point the camera 20 from one real space region to another real space region, and then redirect it back to the original real space region. In such a case, multiple groups of temporally separated frames representing the original real space region are generated, and an accumulated error occurs in the inter-frame projective transformation matrix connecting the respective frames. The present method can reduce the accumulated error between non-consecutive frames representing a common real space region.

[0055] The inter-frame analysis unit 124 may execute any one of the processes of methods 1 to 7, or may execute a combination of some or all of them as long as no contradiction occurs. The inter-frame analysis unit 124 may execute the process of determining a frame group and the process of calculating an inter-frame projective transformation matrix independently or sequentially, or may execute both synchronously. Next, an example of the calculation process of the inter-frame projective transformation matrix will be described. Fig. 19 is a flowchart illustrating the calculation process of the inter-frame projective transformation matrix. Fig. 19 illustrates a case where the step of determining the frame group and the step of calculating the inter-frame projective transformation matrix are synchronized.

[0056] (Step S202) The inter-frame analysis unit 124 initially sets, as a group of consecutive frames, a predetermined number of temporally adjacent frames for which it is fully expected that the image overlap rate of the common area will be equal to or greater than a predetermined ratio from the images of each frame shown in the image data. The inter-frame analysis unit 124 provisionally calculates an inter-frame projective transformation matrix between adjacent frames using method 2 for each frame in the group of consecutive frames. The inter-frame analysis unit 124 then calculates the optical flow from the image of the first frame to the image of the last frame of the group of consecutive frames, and calculates the inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the calculated inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the degree of conformance 1 between the inter-frame projective transformation matrices of adjacent frames in the continuous frame group, and calculates the cumulative value H f-multi The inter-frame analysis unit 124 calculates the calculated cumulative value H f-multiand calculates the compatibility 2 between the optical flow. The inter-frame analysis unit 124 compares the compatibility 1 and the compatibility 2, and if the compatibility 2 is lower than the compatibility 1, performs the process of method 5 on the group of consecutive frames to adjust the inter-frame projective transformation matrix between adjacent frames. Thereafter, the inter-frame analysis unit 124 performs the process of method 6 to further smooth the inter-frame projective transformation matrix between adjacent frames across multiple frames.

[0057] (Step S204) The inter-frame analysis unit 124 provisionally calculates an inter-frame projective transformation matrix from the last frame of the consecutive frame group according to method 2, taking the frame immediately following the consecutive frame group as the next frame. (Step S206) The inter-frame analysis unit 124 calculates the optical flow from the image of the start frame of the consecutive frame group to the image of the next frame, and calculates the inter-frame projective transformation matrix H f-long The inter-frame analysis unit 124 calculates the calculated inter-frame projective transformation matrix H f-long The degree of conformity between the object and the optical flow is calculated.

[0058] (Step S208) The inter-frame analysis unit 124 calculates the total product of the inter-frame projective transformation matrix between adjacent frames in the group of consecutive frames and the inter-frame projective transformation matrix from the last frame of the group of consecutive frames to the next frame, and calculates a cumulative value H f-multi The inter-frame analysis unit 124 calculates the calculated cumulative value H f-multi and the optical flow. The inter-frame analysis unit 124 compares the compatibility 1 with the compatibility 2, and if the compatibility 2 is lower than the compatibility 1, executes the process of method 5 on the final frame and the next frame of the consecutive frames to adjust their inter-frame projective transformation matrices. At this stage, it is not necessary to take into account the actions from the start frame of the consecutive frames to the frame immediately before the final frame. After that, the inter-frame analysis unit 124 calculates the compatibility 2 again.

[0059] (Step S210) The inter-frame analysis unit 124 defines an area in which the start frame and the next frame in the consecutive frame group show a common image as a common area, and calculates the ratio of the defined common area to the entire image as the image overlap rate. The inter-frame analysis unit 124 determines whether the calculated image overlap rate is equal to or lower than a predetermined level. If the image overlap rate is equal to or lower than the predetermined level (step S210 YES), the inter-frame analysis unit 124 proceeds to the process of step S216. In this case, the inter-frame analysis unit 124 determines that the next frame cannot be included in the consecutive frame group immediately before it. This determines one consecutive frame group. If the image overlap rate exceeds the predetermined level (step S210 NO), the inter-frame analysis unit 124 proceeds to the process of step S212.

[0060] (Step S212) The inter-frame analysis unit 124 determines whether or not the matching degree 2 is equal to or lower than a predetermined level. If the matching degree 2 is equal to or lower than the predetermined level (step S212 YES), the process proceeds to step S216. In this case, the inter-frame analysis unit 124 also determines that the next frame cannot be included in the immediately preceding group of consecutive frames. If the matching degree 2 is higher than the predetermined level (step S212 NO), the process proceeds to step S214. (Step S214) The inter-frame analysis unit 124 adds the next frame as the final frame to the group of consecutive frames, and then proceeds to the process of step S204.

[0061] (Step S216) The inter-frame analysis unit 124 searches for an angle of view overlap frame whose angle of view overlaps with the angle of view of the start frame of the determined consecutive frame group from past frames preceding the consecutive frame group. An angle of view overlap frame is a frame relating to an image that represents a common field of view, that is, a frame for which an area ratio of a common area with the start frame is equal to or greater than a certain level and an inter-frame projective transformation matrix is ​​obtained for which the matching rate with the optical flow from the start frame is equal to or greater than a certain level. If an angle of view overlap frame is found (step S216 YES), the process proceeds to step S218. If an angle of view overlap frame is not found (step S216 NO), the process of FIG. 19 ends. (Step S218) The inter-frame analysis unit 124 identifies the angle-of-view overlap frame, and executes the process of method 7 on the intermittent frame group including the frame in the latest consecutive frame group to reduce (minimize) the error of the inter-frame projection matrix belonging to the intermittent frame group. In addition, the process of methods 4 to 6 is used for the consecutive adjacent frames existing between each frame included in the intermittent frame group to adjust these inter-frame projective transformation matrices so that they conform to the inter-frame projective transformation matrices between each frame included in the intermittent frame group. Then, the process of FIG. 19 ends.

[0062] Appendix 1. Use of a homography matrix with limited degrees of freedom Appendix 1-1. About Projection Transformation Matrix The transformation of a point between the coordinates in one coordinate system and the coordinates in another coordinate system, or the translation of a point from one point to another in one coordinate system, or the correspondence between two points, is mathematically expressed as a projective transformation. The matrix that represents this projective transformation is called the projective transformation matrix. For example, the relationship between a point m on a two-dimensional coordinate system and a point m′ after projective transformation is expressed by using a projective transformation matrix H and equation (1).

[0063]

number

[0064] In equation (1), the coordinates of points m and m' are expressed in the homogeneous coordinate system by equations (2) and (3). The projective transformation matrix H is a 3-by-3 matrix expressed by equation (4). λ is a scale coefficient. λ can take any scalar value. Due to this arbitrariness, any element, for example, h 22 The projective transformation matrix H may be normalized by setting the value of x to a constant value other than 0 (for example, 1). Therefore, the degree of freedom of the projective transformation matrix H is one less than the number of elements. The degree of freedom of the projective transformation matrix H exemplified by the expressions (1) and (4) is 8.

[0065]

number

[0066]

number

[0067]

number

[0068] Appendix 1-2. Restrictions on the degrees of freedom of the projective transformation matrix In camera geometry, the transformation from a point M on a three-dimensional real space coordinate system to a point m on a camera image coordinate system is expressed using a projective transformation matrix P as shown in equation (5).

[0069]

number

[0070] In formula (5), the coordinates of points M and m are expressed by formulas (6) and (7), respectively. The projective transformation matrix P is a 3-row, 4-column matrix expressed by formula (8).

[0071]

number

[0072]

number

[0073] The projective transformation matrix P is expressed as the product of element matrices A, F, R, [E -T], which indicate the contribution of element parameters, as shown in formula (8). The element matrix A is determined by the skew distortion s, the aspect ratio ρ, and the coordinates (u0, v0) of the intersection between the virtual image plane of the camera image and the optical axis of the camera, as shown in formula (9). Depending on the specifications of the camera 20, s=0, ρ=1, (u0, v0) may be approximated to the coordinates of the center of the image. In addition, the values ​​of s, ρ, u0, and v0 may be determined in advance using a known camera internal parameter estimation technique. The element matrix F is determined by the focal length f as shown in equation (10). The element matrix R is a rotation matrix with three degrees of freedom. The element matrix R is determined by the orientation of the camera in the real space coordinate system as shown in equation (11). The vector T is the coordinates (t x , t y , y z ) The matrix E is a 3-row, 3-column identity matrix.

[0074]

number

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079] For the positions m0 and m1 on the camera image coordinate system projected in two different frames of point M on the real-time coordinate system, the relationship shown in equation (13) is derived from equations (5) and (8). In equation (13), 0 and 1 indicate either of the two frames. The element matrices F and R and the vector T are variable depending on the frame.

[0080]

number

[0081] If the real space coordinate system is set to match the position and orientation of the camera in frame 0, and the scale of the real space coordinate system is set so that the focal length f0 in frame 0 is f0 = 1, then T0 = [0,0,0] T , R0=E, F0=E. Furthermore, if it is assumed that the camera position is constant on the real space coordinate system during shooting, that is, T0=T1, formula (13) is rearranged as formula (14). Therefore, the projective transformation matrix H from frame 0 to frame 1 is given by formula (15). Since the element matrix A is an internal parameter that can be acquired from the camera 20 by the image processing device 10 or set for the camera 20, the degrees of freedom of the projective transformation matrix H are four in total, which is one degree of freedom of the element matrix F1 and three degrees of freedom of the rotation matrix R1. Furthermore, if no zoom operation is performed during shooting, the element matrix F1 becomes equal to the element matrix F0 (can be regarded as F1=E), so the degrees of freedom of the projective transformation matrix H is further reduced to three. In the projective transformation matrix H with the degrees of freedom restricted in this way, the components due to the parameters with the degrees of freedom restricted and variable change, and the components due to the parameters with the degrees of freedom restricted and fixed do not change.

[0082]

number

[0083]

number

[0084] Appendix 1-3. Quantification of displacement between camera images using a homography matrix with restricted degrees of freedom The projective transformation between frames handled in this application may be expressed with degrees of freedom lower than the original degrees of freedom depending on the shooting conditions of the images. By using a projective transformation matrix with limited degrees of freedom, it is possible to prevent unrealistic displacements of the images from being searched for, and it is possible to stably estimate the projective transformation matrix. When the change in the camera position during shooting can be ignored and the changes in the camera orientation and zoom are allowed, the inter-frame analysis unit 124 can use a projective transformation matrix with four degrees of freedom modeled as in Equation (16).

[0085]

number

[0086] When changes in camera position and zoom during shooting can be ignored and changes in camera orientation are allowed, the inter-frame analysis unit 124 can use a projective transformation matrix with three degrees of freedom modeled as in equation (17).

[0087]

number

[0088] When determining the inter-frame homography matrix, the inter-frame analysis unit 124 calculates the image transition between target frames as an optical flow for each block. Then, the inter-frame analysis unit 124 can estimate the inter-frame homography matrix under the above model using a regression method such as the RANSAC method so as to increase the degree of conformity with the calculated optical flow. Specifically, the inter-frame analysis unit 124 can calculate the inter-frame homography matrix by executing the procedure illustrated in FIG. 20.

[0089] The inter-frame analysis section 124 repeats the processing of steps S302 to S306 a predetermined number of times, and then proceeds to the processing of step S308. (Step S302) The inter-frame analysis unit 124 randomly selects optical flows according to the number of degrees of freedom of the model. For example, when using models with four degrees of freedom and three degrees of freedom, the number of optical flows is two. (Step S304) The inter-frame analysis unit 124 applies a model to the selected optical flow. That is, the inter-frame analysis unit 124 determines parameters of the model so that the motion indicated by the selected optical flow is as close as possible to the motion given by the inter-frame projective transformation matrix calculated using the model.

[0090] (Step S306) The inter-frame analysis unit 124 determines whether the motion indicated by the other optical flows not selected matches the motion given by the inter-frame projective transformation matrix calculated using a model under the determined parameters. The inter-frame analysis unit 124 can determine whether or not the motion matches based on, for example, whether the difference between the two types of motion is equal to or smaller than a predetermined reference value. The inter-frame analysis unit 124 counts the number of other optical flows whose motion matches as the degree of matching. (Step S308) The inter-frame analysis unit 124 determines the inter-frame homography matrix that gives the highest degree of conformance as the regression value, and then ends the process of FIG.

[0091] In addition, when the position of the camera 20 is not changed but the orientation and zoom are changed, and the subject is sufficiently far away from the camera 20, it can be assumed that the image transition between frames is composed of elements of translation (2 degrees of freedom, parameters: Δu, Δv), rotation (1 degree of freedom, parameter α), and enlargement or reduction (1 degree of freedom, parameter f). Therefore, the homography matrix can be approximated as shown in equation (18). In this case, the number of optical flows to be randomly extracted in the regression is two. In addition, when it is assumed that no enlargement or reduction occurs, the parameter f is fixed to 1.

[0092]

number

[0093] The derivation of parameters suitable for the two optical flows is analytically realized using a closed form solution described below instead of parameter optimization, which reduces the computational cost. Coordinates m0 on the camera image system ([u0, v0] T ) to coordinate m1([u1, v1] T ) is rearranged as shown in equations (19) to (22) using the projection transformation matrix H shown in equation (18).

[0094]

number

[0095]

number

[0096]

number

[0097]

number

[0098] The coordinates m0 and m1 before and after the transformation are obtained from each of the two randomly extracted optical flows. 00 , m 10 and the coordinates m before and after the transformation for the other optical flow 1 01 , m 11 are expressed by equations (23) and (24). By substituting equations (23) and (24) into equation (22) and solving them simultaneously, equation (25) is obtained, which is further transformed into equation (26). According to equation (26), the inter-frame analysis unit 124 can determine the enlargement / reduction ratio f, the rotation angle α, and the translation amount (Δu, Δv) from the two optical flows.

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103] On the other hand, if the position of the camera 20 changes during shooting, the degrees of freedom required to express that movement increase by three, resulting in seven degrees of freedom, including one for the focal length and three for the camera orientation. This degree of freedom is close to the eight degrees of freedom of the 3-row, 3-column homography matrix H shown in equation (27), and can be directly applied to this homography matrix H. When applying, it is necessary to extract optical flows at four points. Even in this case, the inter-frame analysis unit 124 can analytically calculate the parameters using a closed-form solution.

[0104]

number

[0105] Appendix 2. Use of projective transformation matrix to represent small movements The projective transformation matrix representing the minute movement refers to the above-mentioned minute transformation matrix ΔH. The minute transformation matrix ΔH is derived by randomly varying at least some element parameters belonging to the camera parameters. In the present application, the minute transformation matrix ΔH is added to the interframe projective transformation matrix H, and the compatibility with the optical flow is confirmed for the interframe projective transformation matrix H after the addition of the minute transformation matrix ΔH, thereby proposing to search for an interframe projective transformation matrix H having a higher compatibility with the optical flow. In a method of searching using a regression method such as RANSAC from the optical flow, the true interframe projective transformation matrix is ​​not guaranteed unless data that is strictly true can be sampled. However, by executing the following method to expand the range of interframe projective transformation matrix candidates to be searched, the compatibility with the optical flow and therefore the accuracy of the interframe projective transformation matrix H can be improved.

[0106] The interframe analysis unit 124 generates the infinitesimal transformation matrix ΔH using a model with the camera parameters as explanatory variables. The interframe analysis unit 124 selects a model having degrees of freedom corresponding to the shooting conditions. The interframe analysis unit 124 can acquire data such as operation information such as zooming, and a sensor signal indicating a moving state or a direction from the camera 20, and determine the shooting conditions based on the acquired data. For example, when the position of the camera 20 does not change and the zoom changes, the interframe analysis unit 124 generates the infinitesimal transformation matrix ΔH using equation (28). Equation (28) corresponds to a model similar to equation (16), and includes an element matrix F1 (degree of freedom 1) and a rotation matrix R1 (degree of freedom 3) as variable factors. Therefore, the degrees of freedom of the infinitesimal transformation matrix ΔH are 4. Note that, when the interframe analysis unit 124 determines that no zoom change occurs, it fixes the element matrix F1 including the focal length to F1=E, and makes only the rotation matrix R1 variable. In equation (28), when F1=E and R1=E, the infinitesimal transformation matrix ΔH becomes a unit matrix, which indicates that no movement of the image occurs.

[0107] The interframe analysis unit 124 generates minute random noise with a value that gives element matrices F1=E, R1=E as a center value for the element parameters to be made variable, and generates a minute transformation matrix ΔH including the element matrices F1, R1 as factors based on the generated random noise. For example, the interframe analysis unit 124 can generate a sequence of values ​​obtained by normalizing each sample value constituting a pseudorandom number sequence generated by a known method so that the sample value falls within a predetermined center value and value range as the minute random noise. The value range is set to be larger than a calculation error (e.g., quantization error, pixel interval) and sufficiently smaller than a typical amount of fluctuation during shooting. Note that, when the interframe analysis unit 124 determines that no zoom change will occur, F1=E, that is, the focal length f is fixed to 1, and only the direction is made variable.

[0108]

number

[0109] The inter-frame analysis unit 124 may generate the infinitesimal transformation matrix ΔH using a model shown in formula (29). The model shown in formula (29) is the same as the model in formula (18), and its degree of freedom is 4. In formula (29), if Δu=0, Δv=0, α=0, and f=1, the infinitesimal transformation matrix ΔH becomes a unit matrix, which indicates that no image movement occurs. Then, infinitesimal random noise with central values ​​of 0, 0, 0, and 1 is generated for each element parameter Δu, Δv, α, and f, respectively, and the infinitesimal transformation matrix ΔH is generated using formula (29) based on the generated random noise. Note that, when the inter-frame analysis unit 124 determines that no zoom change occurs, it fixes the focal length f to 1 and makes only Δu, Δv, and α variable.

[0110]

number

[0111] In addition, when it is already known how to improve the interframe projective transformation matrix H, that is, when an ideal value or a target value is set, the interframe analysis unit 124 calculates the change amount H required for this improvement before performing the minute transformation on the interframe projective transformation matrix H. diff In the case where a pair of two frames related to the inter-frame projective transformation matrix H are adjacent to each other, the inter-frame analysis unit 124 may try to correct the already obtained inter-frame projective transformation matrix H based on the above. diff and the multiplied value is determined as the corrected inter-frame projective transformation matrix H. When the pair of two frames related to the inter-frame projective transformation matrix H is separated in time by two or more frames, the inter-frame analysis unit 124 multiplies the amount of change H diff is distributed to the difference component dH for each pair of adjacent frames, and the inter-frame projective transformation matrix H for each pair of adjacent frames is corrected by multiplying it by the difference component dH for each pair of adjacent frames. For example, the ideal interframe projective transformation matrix H i→i+l f is known, the interframe analysis unit 124 calculates the current value H i→i+l f ' is multiplied by the ideal value to obtain the change H diff Calculate.

[0112]

number

[0113] Then, the inter-frame analysis unit 124 calculates the amount of change H diff is calculated as the difference component dH to be distributed to the interframe projective transformation matrix of each set of l-1 adjacent frames in the frame group. Here, when the interframe projective transformation matrix is ​​derived using a reference frame, the set of adjacent frames from the starting frame to the reference frame is also calculated as the difference component dH. diffFor example, the inter-frame analysis unit 124 can calculate the difference component dH using the model of equation (31). The model of equation (31) is the same as equation (28), and has four degrees of freedom.

[0114]

number

[0115] The inter-frame analysis unit 124 calculates the difference H diff The inter-frame analysis unit 124 divides the difference component dH for each pair of adjacent frames from m0 to m1 as shown in equation (32) as follows: diff Multiplying this by m i The inter-frame analysis unit 124 calculates the total product Π′ of the difference components dH for the number of pairs of adjacent frames in the frame group as shown in equation (34). l Multiplying by dH gives the estimated movement amount m i The inter-frame analysis unit 124 determines the target movement amount m i ' and the estimated movement amount m i A difference component dH for each pair of adjacent frames that minimizes the sum of squared differences of " is searched for. Note that f1 in equation (35) denotes a focal length parameter that constitutes the element matrix F1.

[0116]

number

[0117]

number

[0118]

number

[0119]

number

[0120] Appendix 3. Weighted moving average (smoothing) of the homography matrix over successive frames The weighted moving average corresponds to calculating a weighted sum of element values ​​for each element of inter-frame projective transformation matrices of consecutive adjacent frames, assuming similarity between the matrices. In the example of equation (36), the weight for the target frame is basically (1-2α), and the weights for the previous frame immediately before it and the next frame immediately after it are each set to α. The coefficient α is a coefficient that indicates the degree of the smoothing effect achieved by the moving average. The coefficient α is a real number greater than 0 and less than 0.5. However, since the previous frame is not taken into account for start frame 0 in the frame group, the weight for start frame 0 is set to 1-α. Since the subsequent frame is not taken into account for the final frame l, the weight for final frame l is set to 1-α.

[0121]

number

[0122] The interframe homography matrix H smoothed by the weighted moving average obtained based on Eq. (36) i ′ does not necessarily indicate a weighted average of a geometrically strict projective transformation. Therefore, when performing the moving average, the inter-frame analysis unit 124 uses the inter-frame projective transformation matrix H i ' and the optical flow between the corresponding frames are evaluated, and if the compatibility improves, the inter-frame projective transformation matrix H i ', if the compatibility does not improve, the original interframe projective transformation matrix H i Specifically, the inter-frame analysis unit 124 uses the individual inter-frame projective transformation matrices H i The procedure shown in FIG. 21 is executed for

[0123] (Step S402) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i Based on a series of interframe projective transformation matrices within a predetermined interval including i ' to estimate. (Step S404) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i The moving average value of H i The degree of conformity 1 between ' and the optical flow between the frames is evaluated. (Step S406) The inter-frame analysis unit 124 calculates the inter-frame projective transformation matrix H i The degree of conformity 2 between the optical flow and the frame is evaluated.

[0124] (Step S408) The inter-frame analysis unit 124 compares the conformance level 1 with the conformance level 2, and determines whether or not the conformance level 1 is lower than the conformance level 2. If the conformance level 1 is lower (YES in step S408), the original inter-frame projective transformation matrix H i is adopted, and the process in Fig. 21 ends. If the degree of compatibility 1 does not decrease (NO in step S408), the process proceeds to step S410. (Step S410) The inter-frame analysis unit 124 calculates the moving average value H i ' is adopted to obtain the interframe projection transformation matrix H i Update as. (Step S412) The inter-frame analysis unit 124 judges whether the number of repetitions of the processes of steps S404 to S410 has reached a predetermined number of repetitions. If it is judged that the number has not been reached (step S412 YES), the process returns to step S404. If it is judged that the number has been reached (step S412 NO), the process of FIG. 21 ends.

[0125] The inter-frame analysis unit 124 does not necessarily need to perform the moving average for all inter-frames. The inter-frame analysis unit 124 may perform the moving average for the inter-frame projective transformation matrices for some inter-frames (for example, between any one frame or more and less than l frames in one frame group).

[0126] Appendix 4. Projection transformation matrix between real space plane coordinate system and camera image coordinate system (real space projective transformation matrix) H g Derivation of The coordinate transformation from point M on the real space coordinate system to point m on the camera image coordinate system is expressed by equations (37) to (40) using a projective transformation matrix P with 3 rows and 4 columns.

[0127]

number

[0128]

number

[0129]

number

[0130]

number

[0131] If it is assumed that the position of point M exists on the plane z=0 in real space, the relationships shown in equations (37) to (40) can be rearranged as shown in equations (41) and (42).

[0132]

number

[0133]

number

[0134] In equation (42), by omitting the third column of the projection transformation matrix P and the z coordinate of point m, a 3-row, 3-column real space projection transformation matrix H g is derived.

[0135]

number

[0136]

number

[0137]

number

[0138] In equation (45), p 23 When normalized as =1, the relationships shown in equations (46) and (47) are derived.

[0139]

number

[0140]

number

[0141] Here, for n points with known coordinates arranged on a real space plane, the image coordinate values ​​(u k , v k ) (k=0, …, n-1) are obtained, equation (47) can be expressed as the simultaneous equations in equation (48). If the matrix in the first term on the left side of equation (48) is B as in equation (49) and the vector on the right side is C as in equation (50), equation (48) can be rearranged as in equation (51). Equation (51) is p 23 = 1, and convert matrix B and vector C into real space projection transformation matrix H g is calculated. T B) has an inverse matrix, its rank must be 8. Therefore, if there are four or more known coordinate points, the real space projection transformation matrix H g It is shown that it is possible to calculate

[0142]

number

[0143]

number

[0144]

number

[0145]

number

[0146] In addition, the real space projection transformation matrix H g The calculation of may be performed by applying a known method. For example, when six or more real space coordinate known points, including points not on a plane, can be identified in the standard frame, the projection transformation matrix P is calculated from the relationship between the real space coordinate values ​​of these real space coordinate known points and the camera image coordinate values ​​of the image corresponding points. Based on the element values ​​of this P and equation (45), H g Calculate.

[0147] Then, as shown in equation (54) derived from equations (52) to (53), the real space projection transformation matrix H g For the interframe projection transformation matrix H f By multiplying by , the real space projection transformation matrix H g Therefore, the real space projection transformation matrix H g and for every pair of frames, the interframe projection matrix H f If the matrix H g is obtained.

[0148]

number

[0149]

number

[0150]

number

[0151] Appendix 5. Derivation of the projective transformation matrix P between the real space coordinate system and the camera image coordinate system Appendix 5-1. Calculation of focal length and external camera parameters Real space projection transformation matrix H g is expressed as a product of element matrices A, F, R, etc., using the camera parameters, as shown in equation (55). The relationship shown in equation (56) holds for the first and second columns on both the left and right sides of equation (55). Then, the elements of the first and second columns of the rotation matrix R are expressed as column vector r1 T , r2 T Then, equation (56) is transformed into equation (57). Note that in this example, the parameters s and ρ constituting the element matrix A are approximated as s = 0 and ρ = 1, but these parameters may be set to predetermined values.

[0152]

number

[0153]

number

[0154]

number

[0155] According to the nature of the rotation matrix R, the column vector r0 T , r1 T r0 T r0=r1 T r1, r0 TThe relationship r1 = 0 is satisfied. According to this relationship, equation (58) can be obtained from equation (57). By transferring the focal length f to the left side of equation (58), two equations (59) related to f are obtained.

[0156]

number

[0157]

number

[0158] The calibration unit 130 may calculate the focal length f by using either one of the two equations in formula (59). However, the focal length f may differ for each equation due to the influence of errors related to the processing up to this point. Therefore, the calibration unit 130 may calculate the average value of both as the focal length f. For example, the calibration unit 130 may calculate the focal length f by using the average value of f obtained from each of the two equations as shown in formula (60). 2 The square root of the average value of these may be calculated as the focal length f.

[0159]

number

[0160] The calibration unit 130 can calculate the rotation matrix R and the element matrix T according to the equations (61) to (67) using the obtained focal length f. The rotation matrix R indicates the orientation of the camera 20. The element matrix T indicates the position of the camera 20. The element matrix F is a matrix formed from the focal length f of the camera 20.

[0161]

number

[0162]

number

[0163]

number

[0164]

number

[0165]

number

[0166]

number

[0167]

number

[0168] Appendix 5-2. Provisional calculation of camera position and recalculation of focal length and camera direction The focal length f and the camera position calculated by the method described in Supplementary Note 5-1 may contain errors and may differ greatly from the true values. On the other hand, the error in the camera orientation and the direction of the camera optical axis vector is relatively small. Also, the error in the camera position tends to occur in the direction of the optical axis vector, which is related to the error in the focal length. Therefore, when the camera position is considered not to change during shooting, the calibration unit 130 provisionally calculates a candidate camera position under the assumption that the camera position exists on the optical axis (see FIG. 17). Here, any coordinate on the optical axis is formulated as being at a position T+kr^2 that is k times the displacement amount indicated by the third row vector r^2 (corresponding to a vector parallel to the camera optical axis) of the rotation matrix R from the camera position T. The distance D between the coordinate T0 of the candidate camera position and the camera optical axis is given by Equation (68).

[0169]

number

[0170] The calibration unit 130 performs calibration using D or D for all frames in a frame group or for all frames across a frame group that share a common region. 2 The sum of these is calculated, and T0 that minimizes this sum is searched for as the coordinates of the camera position candidate. The calibration unit 130 calculates the focal length f and the rotation matrix R in each frame using the calculated coordinates T0 of the camera position candidate. The formula for calculating the focal length f is derived as follows. 0x , T 0y , T 0z ] T ) is substituted into equation (69), which is rearranged into equation (70). For the rotation matrix R, T Since the relationship R=E holds, equation (70) is further modified in sequence to equations (71) to (74). In this example, parameters s and ρ constituting element matrix A are approximated as s=0 and ρ=1, but these parameters may be set to values ​​determined in advance. Noting that equation (74) is a symmetric matrix, six equations related to focal length f can be obtained as shown in equation (75). By eliminating coefficient λ while satisfying the relationships shown in each equation in equation (75), equation (76) is obtained.

[0171]

number

[0172]

number

[0173]

number

[0174]

number

[0175]

number

[0176]

number

[0177]

number

[0178]

number

[0179] The calibration unit 130 may calculate the focal length f by using any one of the five equations shown in equation (76). However, the focal length f calculated for each equation may differ due to the influence of errors related to the processing up to this point. Therefore, the calibration unit 130 may calculate the focal length f by using any two to four equations or all five equations in equation (76). 2 The square root of the average value of these may be calculated as the focal length f. The calibration unit 130 can calculate the rotation matrix R according to equations (77) to (82) using the calculated focal length f.

[0180]

number

[0181]

number

[0182]

number

[0183]

number

[0184]

number

[0185]

number

[0186] Appendix 5-3. Bundle adjustment In the process up to Appendix 5-2, the real space projection transformation matrix H g The camera position candidate T0, the focal length f for each frame, and the camera orientation R provisionally calculated based on are algebraically appropriate solutions, but are not considered to be likely parameter values ​​in terms of camera geometry. The calibration unit 130 uses the provisionally calculated values ​​as initial values ​​and searches for camera parameters that minimize the reprojection error for the real space coordinate known points. This process is called bundle adjustment. The procedure of bundle adjustment will be described below. A known point M on a plane in real space coordinate system i ' and the coordinates of the reprojected points m i The relationship between M and M' is given by equations (83) and (84). However, it is assumed that four or more image corresponding points have already been determined in each frame. Note that the aggregated image corresponding points may be located outside the angle of view of the image in the frame, but these may be used in the bundle adjustment process, or only a portion of them may be used, or only image corresponding points determined within the angle of view may be used without using these. In equation (84), M i ' denotes the real space coordinates of the real space coordinates known point detected in frame i. Also, m i is the real space coordinate known point M i ' corresponds to image corresponding point m i The coordinates of are shown.

[0187]

number

[0188]

number

[0189]

number

[0190] The calibration unit 130 sets the error between the image corresponding point and the reprojected point as the reprojection error e, sets the sum of the squared value of the reprojection error shown in Equation (86) for each real space coordinate known point and each frame in the frame group as the error index, and searches for the coordinate T0 of the camera position, the focal length f, and the rotation matrix R that minimizes the error index for some or all of the frames (minimization of the reprojection error). The calibration unit 130 can use the coordinate T0, the focal length f, and the rotation matrix R of the camera position candidate obtained in the process up to Appendix 5-2 as the initial values. Note that the calibration unit 130 may set the coordinate T0 of the camera position obtained for some frames (for example, the reference frame) as a default value, and use the focal length and rotation matrix R provisionally calculated based on this coordinate T0 as the initial values ​​to search for the focal length f and the rotation matrix R that minimizes the error index for the other frames.

[0191]

number

[0192] Appendix 5-4. Derivation of the projective transformation matrix P between the real space coordinate system and the camera image coordinate system and the real space projective transformation matrix H g , the inter-frame projection transformation matrix H f Update The calibration unit 130 uses the coordinates T0 of the camera position, the focal length f, and the rotation matrix R optimized in the process of Appendix 5-3 to calculate the projective transformation matrix P according to equation (87). The calibration unit 130 also performs calculations based on equations (88) and (89) so that the parameters match the optimized parameters, and calculates the real space projective transformation matrix H g and update the interframe projection transformation matrix H f may be updated.

[0193]

number

[0194]

number

[0195]

number

[0196] Appendix 5-5. Variability of assumptions In the examples from Supplementary Note 5-1 to Supplementary Note 5-4, it is assumed that the camera parameters constituting the element matrix A are s=0, ρ=1, and u0 and v0 are default values, but this is not limited to the above. Some or all of s, ρ, u0, and v0 may be measured values ​​measured in advance. In addition, in the bundle adjustment described in Supplementary Note 5-3, some or all of these parameters may be included in the optimization target as variable variables. When movement of the camera 20 is detected, the camera position T0 assumed not to move may be made variable for each frame, and the step described in Supplementary Note 5-2 may be omitted. In addition, when a zoom operation is not performed, the focal length f assumed to be different for each frame may be assumed to be constant between frames. The calculation processing unit 120 may determine the assumption conditions to be applied based on the operation information acquired from the camera 20, various data indicating the moving state or direction, or the like, or the operation signal indicating the operation on the camera 20 input from the operation input unit 160.

[0197] As described above, the image processing device 10 according to the present embodiment calculates an inter-frame projective transformation matrix indicating the transition of images between frames, identifies image corresponding points corresponding to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames, and aggregates and determines image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on the inter-frame projective transformation matrix, thereby identifying image corresponding points. The image processing device 10 calculates a real space projective transformation matrix of the reference frame based on the real space coordinate known points and the aggregated image corresponding points, calculates a real space projective transformation matrix of the other frames based on the inter-frame projective transformation matrix and the real space projective transformation matrix between the images of the reference frame and the other frames, determines image corresponding points in the other frames based on the inter-frame projective transformation matrix, and adjusts at least a part of the camera parameters for each frame so as to reduce the difference between the estimated value of the image corresponding point corresponding to the real space coordinate known point estimated based on the camera parameters constituting the real space projective transformation matrix and the image corresponding point corresponding to the real space coordinate known point.

[0198] According to this configuration, a real space projective transformation matrix indicating a relationship between an image corresponding point corresponding to a real space coordinate known point in a reference frame that is one of the multiple frames and an inter-frame projective transformation matrix indicating an image transition between frames are determined, and a real space projective transformation matrix for each frame is determined based on the determined real space projective transformation matrix and inter-frame projective transformation matrix of the reference frame. Then, at least a part of the camera parameters (e.g., focal length) is determined by adjusting so as to reduce the difference between the image corresponding point for each frame and the image corresponding point estimated from the corresponding real space coordinate known point. Therefore, camera calibration can be realized under conditions with fewer constraints. For example, camera calibration can be realized for each frame even if a marker always set at a preset known point of real space coordinates does not appear in the captured image. Furthermore, in camera calibration, even in a situation where the field of view changes due to zooming, rotation, etc., the camera parameters that indicate the situation can be estimated. This makes it easier to perform image analysis in a wider range of shooting conditions.

[0199] This embodiment may be implemented as follows. The image processing device 10 may determine an optical flow indicating image movement for each part in the first frame that starts from the first frame and ends at a part in the second frame that corresponds to the starting point, and calculate an inter-frame projective transformation matrix for a pair of the first and second frames based on the determined optical flow.

[0200] The image processing device 10 may search for at least some camera parameters that indicate a transformation from the origin to an end point corresponding to the origin for each set of a predetermined number of origins, and may search for an inter-frame projective transformation matrix that improves the degree of fit based on the optical flow that matches the end point to the origin, and the estimated value of the end point estimated from the inter-frame projective transformation matrix based on the camera parameters searched for the origin.

[0201] The image processing device 10 may calculate a first inter-frame projective transformation matrix which is an inter-frame projective transformation matrix for a pair of a first frame and a third frame which is a frame separated from the first frame by a predetermined time or more, calculate a second inter-frame projective transformation matrix which is an inter-frame projective transformation matrix for a pair of the third frame and a second frame which is a frame adjacent to the first frame, and calculate an inter-frame projective transformation matrix for the pair of the first frame and the second frame using the first inter-frame projective transformation matrix and the second inter-frame projective transformation matrix.

[0202] The image processing device 10 may execute an inter-frame projective transformation matrix search process to search for an inter-frame projective transformation matrix for at least some pairs of adjacent frames such that the inter-frame projective transformation matrix for a pair of the first and last frames in a frame group consisting of a plurality of frames matches the product of the inter-frame projective transformation matrices for each pair of adjacent frames included in the frame group.

[0203] The image processing device 10 calculates inter-frame projective transformation matrices for at least a part of sets of frames among inter-frame projective transformation matrices for sets of frames included in a frame group made up of a plurality of frames, as follows: The above Camera parameters of element parameters that are elements of At least some A small transformation matrix that corresponds to the random small fluctuations of and executing the inter-frame projective transformation matrix search process using the updated inter-frame projective transformation matrix.

[0204] The image processing device 10 may calculate a moving average value of the inter-frame projective transformation matrix for each pair of frames, calculate a correction value of the inter-frame projective transformation matrix based on a difference between the inter-frame projective transformation matrix and the moving average value, evaluate a first compatibility which is a compatibility between the inter-frame projective transformation based on the correction value of the inter-frame projective transformation matrix and an optical flow from the origin block to the corresponding block, evaluate a second compatibility which is a compatibility between the inter-frame projective transformation and the optical flow, and update the inter-frame projective transformation based on the correction value of the inter-frame projective transformation matrix as the inter-frame projective transformation matrix when the first compatibility is not lower than the second compatibility.

[0205] The image processing device 10 may define, as a frame group, a plurality of consecutive frames in which the ratio of a common area representing a common field of view is equal to or greater than a predetermined ratio.

[0206] Another aspect of this embodiment may be a program for causing a computer to function as the image processing device 10.

[0207] Another aspect of this embodiment may be an image processing system S1 including a camera 20 that captures an image and an image processing device 10.

[0208] Another aspect of this embodiment may be an image processing method in an image processing device 10, which calculates an inter-frame projective transformation matrix that indicates a transition of images between frames, identifies image corresponding points that correspond to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames, aggregates and determines image corresponding points in an image of a reference frame that is one of the frames of the frame group based on the inter-frame projective transformation matrix, calculates a real space projective transformation matrix of the reference frame based on the known real space coordinate points and the aggregated image corresponding points, calculates a real space projective transformation matrix of the reference frame based on the inter-frame projective transformation matrix between the reference frame and the images of the other frames and the real space projective transformation matrix of the reference frame, determines image corresponding points in the other frames based on the inter-frame projective transformation matrix, and determines the camera parameters so as to reduce a difference between the image corresponding points for each frame and an estimated value of the image corresponding point estimated based on the camera parameters from the known real space coordinate points that correspond to the image corresponding points.

[0209] In addition, a part of the image processing device 10 in the above-mentioned embodiment, for example, the arithmetic processing unit 120, may be realized by a computer. In that case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the control function. In addition, the "computer system" here refers to a computer system built into the image processing device 10, and includes hardware such as an OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the "computer-readable recording medium" may also include a medium that dynamically holds a program for a short time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned function, and may further be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system. Also, a part or all of the image processing device 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image processing device 10 may be individually made into a processor, or a part or all of the blocks may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on that technology may be used.

[0210] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0211] S1...image processing system, 10...image processing device, 20...camera, 30...display, 120...arithmetic processing unit, 122...image acquisition unit, 124...frame-to-frame analysis unit, 126...coordinate known point identification unit, 128...real space image-to-image analysis unit, 130...calibration unit, 132...output processing unit, 140...storage unit, 150...input / output unit, 160...operation input unit

Claims

1. Calculate an inter-frame projective transformation matrix that indicates a transition of an image between frames; Identifying image corresponding points corresponding to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames; determining the image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on an inter-frame projective transformation matrix; Calculating a real space projection transformation matrix of the reference frame based on the real space coordinate known points and the image corresponding points; calculating a real space projective transformation matrix of the other frame based on an inter-frame projective transformation matrix between the reference frame and the image of the other frame and the real space projective transformation matrix; determining image corresponding points in another frame based on the inter-frame projective transformation matrix; At least a part of the camera parameters is adjusted for each frame so that a difference between an estimated value of an image corresponding point corresponding to the real space coordinate known point, which is estimated based on the camera parameters constituting the real space projection transformation matrix, and the image corresponding point corresponding to the real space coordinate known point is reduced. Image processing device.

2. determining an optical flow indicating an image movement with an end point being a part in the second frame corresponding to the part in the first frame as an origin; Calculating the inter-frame projective transformation matrix related to the pair of the first frame and the second frame based on the optical flow. The image processing device according to claim 1 .

3. Finding at least some camera parameters for each set of a predetermined number of origin points that describe a transformation from the origin points to a corresponding end point; An estimated value of the end point is estimated from an inter-frame projective transformation matrix based on the camera parameters searched for the starting point, and an inter-frame projective transformation matrix that improves the degree of conformance based on the optical flow that conforms the end point to the starting point is searched for. The image processing device according to claim 1 .

4. calculating a first inter-frame projective transformation matrix which is an inter-frame projective transformation matrix for a pair of a first frame and a third frame which is a frame separated from the first frame by a predetermined time or more, and calculating a second inter-frame projective transformation matrix which is an inter-frame projective transformation matrix for a pair of the third frame and a second frame which is a frame adjacent to the first frame, Calculating the inter-frame projective transformation matrix related to the pair of the first frame and the second frame using the first inter-frame projective transformation matrix and the second inter-frame projective transformation matrix. The image processing device according to claim 1 .

5. An interframe projective transformation matrix search process is performed to search for an interframe projective transformation matrix for at least some of the sets of frames such that the interframe projective transformation matrix of the first frame and the last frame of a frame group consisting of a plurality of frames matches the product of the interframe projective transformation matrices for each set of frames included in the frame group. The image processing device according to claim 1 .

6. updating the inter-frame projective transformation matrix by applying a minute transformation matrix corresponding to a random minute variation of at least a part of element parameters that are elements of the camera parameters to the inter-frame projective transformation matrix for at least a part of the frame sets of the inter-frame projective transformation matrix for each frame set included in a frame group consisting of a plurality of frames; The inter-frame projective transformation matrix search process is executed using the updated inter-frame projective transformation matrix. The image processing device according to claim 5 .

7. For each set of frames Calculating a moving average value of the inter-frame projective transformation matrix; A first degree of matching is evaluated, which is a degree of matching between the motion of an image due to inter-frame projective transformation based on the inter-frame projective transformation matrix and an optical flow. evaluating a second degree of conformance between the motion of an image due to inter-frame projective transformation based on the moving average value and the optical flow; When the second degree of conformance does not decrease below the first degree of conformance, the moving average value is updated as the inter-frame projective transformation matrix. The image processing device according to claim 5 .

8. A plurality of consecutive frames in which the ratio of a common area representing a common field of view is equal to or greater than a predetermined ratio is defined as the frame group. The image processing device according to claim 1 .

9. To your computer A program for causing the image processing device according to claim 1 to function.

10. A camera for capturing the image; The image processing device according to claim 1 , Image processing system.

11. 1. A method in an image processing device, comprising: Calculate an inter-frame projective transformation matrix that indicates a transition of an image between frames; Identifying image corresponding points corresponding to four or more real space coordinate known points, which are points whose real space coordinates are known on a real space plane, in any frame of a frame group consisting of a plurality of frames; determining the image corresponding points in an image of a reference frame, which is one of the frames of the frame group, based on an inter-frame projective transformation matrix; Calculating a real space projection transformation matrix of the reference frame based on the real space coordinate known points and the image corresponding points; calculating a real space projective transformation matrix of the other frame based on an inter-frame projective transformation matrix between the reference frame and the image of the other frame and the real space projective transformation matrix; determining image corresponding points in another frame based on the inter-frame projective transformation matrix; At least a part of the camera parameters is adjusted for each frame so that a difference between an estimated value of an image corresponding point corresponding to the real space coordinate known point, which is estimated based on the camera parameters constituting the real space projection transformation matrix, and the image corresponding point corresponding to the real space coordinate known point is reduced. Image processing methods.

Images