Information processing device, information processing method, and camera

Abstract

The present technology relates to an information processing device, an information processing method, and a camera with which it is possible to accurately and easily detect the center in a process for stabilizing a camera. The information processing device comprises a detection unit for detecting a stabilization center, which is the center in the stabilization process, on the basis of the position of a prescribed feature point in a plurality of photographed images obtained by photographing in a plurality of orientations differing in roll-direction angle in a state in which the camera has executed the stabilization process. The present technology can be applied, for example, to a system for adjusting the principal point of an internal parameter of a camera.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, an information processing method, and a camera, and particularly to an information processing device, an information processing method, and a camera that are suitable for use when detecting the center of stabilization processing. [Background technology]

[0002] Conventionally, a method has been proposed for correcting image stabilization using internal camera parameters in a camera equipped with an image sensor, a lens, and a gyro sensor (see, for example, Patent Document 1). In the method of Patent Document 1, the smaller the difference between the principal point (Cx, Cy), which is one of the camera's internal parameters, and the center of stabilization processing (hereinafter referred to as the stabilization center), the higher the accuracy of image stabilization. The stabilization center is the intersection of the central axis (rotation axis) of stabilization processing and the camera's image coordinate system.

[0003] Here, stabilization refers to a process that reduces the effect of camera shake on an image by using electronic image stabilization (EIS), or a combination of electronic image stabilization and optical image stabilizer (OIS), or mechanical image stabilizer (MIS). This method, for example, converts the shape of the captured image in accordance with the rotation of the camera.

[0004] Methods for transforming image shapes include, for example, congruent transformation, similarity transformation, affine transformation, and projective transformation. Congruent transformation is a transformation method that only allows image rotation, and by matching the rotation angle, the images before and after the transformation will match perfectly. Similarity transformation is a transformation method that allows for enlargement and reduction in addition to congruent transformation. Affine transformation is a transformation method that allows for diamond deformation in addition to congruent and similarity transformations. And projective transformation is a transformation method that adds all the functions of congruent transformation, similarity transformation, and affine transformation and also allows for transformation into any quadrangle.

[0005] On the other hand, the principal point (Cx, Cy) of the internal parameters of the camera is set, for example, to the intersection of the optical axis of the lens and the image coordinate system of the camera. More specifically, for example, it is set to the center of the captured image captured by the camera.

[0006] However, during the camera assembly process, errors in lens installation, image sensor mounting, gyro chip mounting, etc. occur, causing the central axis of stabilization processing to vary from camera to camera, and therefore the central axis of stabilization processing and the optical axis of the lens do not necessarily coincide.

[0007] Furthermore, for example, when a camera is worn while playing sports or when the camera is incorporated into a factory assembly robot, the camera may be subjected to strong shocks or vibrations, causing distortion in the lens or housing, which may shift the central axis of the stabilization process. In this case, it becomes necessary to remeasure the stabilization center and correct the principal point (Cx, Cy) of the camera's internal parameters.

[0008] In response to this, Non-Patent Document 1 proposes a camera calibration method. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] International Publication No. 2017 / 014071 [Non-patent literature]

[0010] [Non-Patent Document 1] Zhang, Z. “A Flexible New Technique for Camera Calibration.” IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 22, No. 11, 2000, pp. 1330-1334 Summary of the Invention [Problem to be solved by the invention]

[0011] However, the calibration method described in Non-Patent Document 1 is very complicated.

[0012] The present technology has been developed in light of these circumstances, and aims to enable accurate and easy detection of the center of camera stabilization processing, thereby improving the accuracy of camera shake correction. [Means for solving the problem]

[0013] An information processing device according to a first aspect of the present technology includes a detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of predetermined feature points in multiple captured images obtained by capturing images in multiple postures with different roll direction angles while the camera is performing a stabilization process.

[0014] The information processing method according to the first aspect of the present technology detects a stabilization center, which is the center of the stabilization process, based on the positions of predetermined feature points in multiple captured images obtained by capturing images in multiple postures with different roll angle angles while the camera is performing a stabilization process.

[0015] In a first aspect of the present technology, a stabilization center, which is the center of the stabilization process, is detected based on the positions of predetermined feature points in multiple captured images obtained by capturing images in multiple postures with different roll direction angles while the camera is performing a stabilization process.

[0016] A camera according to a second aspect of the present technology includes a lens, an image sensor, a motion sensor, a stabilization processing unit that performs stabilization processing on captured images obtained by the image sensor based on measurement values ​​of the motion sensor, a detection unit that detects a stabilization center that is the center of the stabilization processing based on positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll direction angles while the stabilization processing is being performed, and detects a principal point of internal parameters based on the detection result of the stabilization center, and a parameter setting unit that sets the detected principal point as a parameter to be used for the stabilization processing.

[0017] In a second aspect of the present technology, a stabilization process is performed on a captured image obtained by an image sensor based on measurement values ​​of a motion sensor, a stabilization center that is the center of the stabilization process is detected based on positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll direction angles while the stabilization process is being performed, a principal point of internal parameters is detected based on the detection result of the stabilization center, and the detected principal point is set as a parameter to be used in the stabilization process. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram showing an embodiment of a measurement system to which the present technology is applied. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of the hardware configuration of a camera. [Figure 3] FIG. 1 is a block diagram showing a first embodiment of a camera and an external terminal. [Figure 4] FIG. 10 is a block diagram showing a second embodiment of a camera and an external terminal. [Figure 5] FIG. 10 is a block diagram showing a third embodiment of a camera and an external terminal. [Figure 6] FIG. 10 is a block diagram showing a fourth embodiment of the camera. [Figure 7] FIG. 1 is a block diagram showing a first embodiment of a rotation mechanism. [Figure 8] FIG. 10 is a block diagram showing a second embodiment of a rotation mechanism. [Figure 9] FIG. 10 is a plan view showing a third embodiment of a rotation mechanism. [Figure 10] FIG. 10 is a block diagram showing a third embodiment of a rotation mechanism. [Figure 11] FIG. 10 is a plan view showing a fourth embodiment of a rotation mechanism. [Figure 12] FIG. 10 is a block diagram showing a fourth embodiment of a rotation mechanism. [Figure 13] 10 is a diagram showing an example in which the center of the lens and the central axis of the rotation mechanism overlap. FIG. [Figure 14] 10A and 10B are diagrams illustrating an example of the locus of feature points in a captured image when the center of the lens and the central axis of the rotation mechanism overlap. [Figure 15] 10A and 10B are diagrams illustrating an example in which the center of the lens and the central axis of the rotation mechanism are misaligned. [Figure 16] 10A and 10B are diagrams illustrating an example of the locus of feature points in a captured image when the center of the lens and the central axis of the rotation mechanism are misaligned. [Figure 17] FIG. 10 is a diagram showing an example in which an XY stage is used as a mounting jig. [Figure 18] FIG. 2 is a diagram showing an example of the size of each part of a camera. [Figure 19] 10A and 10B are diagrams illustrating the positional relationship between a camera coordinate system, an image coordinate system, and a central axis. [Figure 20] 10A and 10B are diagrams for explaining a method for calculating the amount of deviation between the center and the central axis of the camera coordinate system and the image coordinate system. [Figure 21] FIG. 10 is a diagram illustrating a configuration example of an angle jig. [Figure 22] FIG. 10 is a diagram illustrating a configuration example of an angle jig. [Figure 23] FIG. 10 is a diagram showing an example in which an angle jig and an XY stage are combined. [Figure 24] FIG. 10 is a diagram showing an example in which a board showing a pattern including a single feature point is used as a measurement object. [Figure 25]FIG. 10 is a diagram showing an example in which a board showing a pattern including a plurality of feature points is used as a measurement object. [Figure 26] FIG. 10 is a diagram showing an example in which an external object is used as a measurement target. [Figure 27] 10 is a flowchart illustrating a calibration process. [Figure 28] FIG. 10 is a diagram showing a display example of a captured image. [Figure 29] 1A and 1B are diagrams illustrating examples of a mesh in a normal camera coordinate system and a mesh in a camera coordinate system that reflects lens distortion. [Figure 30] 10A and 10B are diagrams showing examples of a photographed image and a cross mark; [Figure 31] FIG. 10 is a diagram showing an example of displaying feature points of a photographed image. [Figure 32] FIG. 10 is a diagram illustrating an example of a trajectory of feature points in a captured image. [Figure 33] FIG. 10 is a diagram illustrating a stabilization process. [Figure 34] FIG. 10 is a diagram showing a display example of a cut-out image. [Figure 35] FIG. 10 is a diagram illustrating an example of a trajectory of feature points in a cut-out image. [Figure 36] FIG. 10 is a diagram showing an example of a trajectory of feature points in a cut-out image. [Figure 37] 1 is a graph showing an example of angular velocity measurements of a motion sensor during a period of quiescence. [Figure 38] FIG. 10 is a diagram for explaining a method for detecting a stabilization center. [Figure 39] FIG. 10 is a diagram for explaining a method for detecting a stabilization center. [Figure 40] FIG. 1 is a block diagram illustrating an example of the configuration of a computer. [Figure 41] FIG. 10 illustrates an example of use of the system for setting the principal point of the internal parameters. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present technology will be described in the following order. 1. Embodiment 2. Variations 3.Other

[0020] <<1. Embodiment>> <Configuration example of measurement system 1> FIG. 1 is a block diagram showing an embodiment of a measurement system 1 to which the present technology is applied.

[0021] The measurement system 1 includes a camera 11, an external terminal 12, a rotation mechanism 13, and a measurement object 14. The measurement system 1 is a system that detects the stabilization center of the camera 11 and sets the principal point (Cx, Cy) of the internal parameters of the camera 11 based on the detection result of the stabilization center.

[0022] In the following description, unless otherwise specified, the principal point of the camera 11 does not refer to the principal point of the optical center of the camera 11, but to the principal point (Cx, Cy) of the internal parameters of the camera 11.

[0023] The camera 11 is attached to a rotation mechanism 13 and photographs an object 14 to be measured.

[0024] The external terminal 12 is configured by, for example, a general-purpose terminal such as a smartphone, a tablet, or a PC (Personal Computer), or a dedicated terminal for camera calibration, etc. The external terminal 12 communicates with the camera 11 by wireless communication or wired communication.

[0025] The rotation mechanism 13 is a mechanism that rotates the camera 11 in a roll direction around a rotation axis AR1. Here, the roll direction of the camera 11 is the direction of rotation around the optical axis of the lens 103 (the Z axis of the camera coordinate system).

[0026] The measurement object 14 is an object that provides feature points used to detect the stabilization center of the camera 11. The camera 11 and the measurement object 14 are placed opposite to the camera 11 so that the feature points of the measurement object 14 do not go outside the viewing angle Φ1 of the camera 11 when the camera 11 rotates around the rotation axis AR1.

[0027] Hereinafter, the distance between the camera 11 and the measurement object 14 is defined as L.

[0028] <Configuration example of camera 11> FIG. 2 shows a schematic diagram of an example of the hardware configuration of the camera 11.

[0029] The camera 11 includes a housing 101, a support substrate 102, a lens 103, an image sensor 104, a motion sensor 105, a DSP (Digital Signal Processor) 106, and a stabilization chip 107.

[0030] The support substrate 102 is placed in the housing 101 so as to be substantially perpendicular to the optical axis AC 1 of the lens 103 .

[0031] The lens 103 is installed at a predetermined position in the housing 101 and forms an image of the subject on the light receiving surface of the image sensor 104 .

[0032] The image sensor 104 is placed on the support substrate 102 so as to be approximately perpendicular to the optical axis AC1 of the lens 103. The image sensor 104 captures an image of the subject formed by the lens 103, and generates a captured image.

[0033] The motion sensor 105 is installed on the surface of the support substrate 102 opposite to the surface on which the image sensor 104 is installed. The motion sensor 105 includes, for example, a gyro sensor and an acceleration sensor, and detects the angular velocity and acceleration of the camera 11.

[0034] The DSP 106 is installed on the inner surface of the housing 101. The DSP 106 performs various types of image processing on the captured image.

[0035] The stabilization chip 107 is installed on the inner surface of the housing 101. The stabilization chip 107 performs stabilization processing on the captured image.

[0036] <Example of functional configuration of camera 11 and external terminal 12> FIG. 3 shows an example of the functional configuration of the camera 11 and the external terminal 12 shown in FIG.

[0037] In addition to the above-described lens 103, image sensor 104, and motion sensor 105, camera 11 includes a parameter setting unit 121, a mesh setting unit 122, an offset setting unit 123, a stabilization processing unit 124, a UI (User Interface) processing unit 125, a signal processing unit 126, an image cropping unit 127, and an I / O (Input / Output) interface unit 128. Image sensor 104, parameter setting unit 121, mesh setting unit 122, offset setting unit 123, stabilization processing unit 124, UI processing unit 125, signal processing unit 126, image cropping unit 127, and I / O interface unit 128 are connected to each other via a bus.

[0038] The image sensor 104 supplies captured image data corresponding to the captured image to a stabilization processing unit 124 and a signal processing unit 126 .

[0039] The parameter setting unit 121 receives input information including internal parameters of the camera 11 input at the external terminal 12, and principal point information indicating the detection result of the principal point of the camera 11 by the detection unit 155 of the external terminal 12, from the external terminal 12 via the I / O interface unit 128. The parameter setting unit 121 sets internal parameters of the camera 11 based on the input information and the principal point information. The parameter setting unit 121 supplies parameter information related to the set internal parameters to the stabilization processing unit 124. The parameter setting unit 121 also transmits the parameter information to the external terminal 12 via the I / O interface unit 128.

[0040] The mesh setting unit 122 receives mesh information about the mesh in the camera coordinate system generated by the mesh creation unit 153 of the external terminal 12 from the external terminal 12 via the I / O interface unit 128. The mesh setting unit 122 stores the mesh information to set the mesh in the camera coordinate system. The mesh setting unit 122 also supplies the mesh information to the image cropping unit 127.

[0041] The offset setting unit 123 receives offset information indicating the offset value of the motion sensor 105 adjusted by the offset adjustment unit 154 of the external terminal 12 from the external terminal 12 via the I / O interface unit 128. The offset setting unit 123 sets the offset value of the motion sensor 105 based on the offset information.

[0042] The offset setting unit 123 generates motion sensor information indicating the angular velocity and acceleration of the camera 11 measured by the motion sensor 105, and supplies the generated information to the stabilization processing unit 124. The offset setting unit 123 also transmits the motion sensor information to the external terminal 12 via the I / O interface unit 128.

[0043] The stabilization processing unit 124 is realized by, for example, the stabilization chip 107 in Fig. 2. The stabilization processing unit 124 performs stabilization processing on the captured image based on the parameter information and motion sensor information. As described above, the stabilization processing is processing (homography processing) that performs projective transformation on the captured image in accordance with the rotation of the camera 11. The stabilization processing unit 124 supplies the signal processing unit 126 with captured image data corresponding to the captured image after the stabilization processing.

[0044] The UI processing unit 125 executes processing related to the user interface of the camera 11.

[0045] The signal processing unit 126 is realized by, for example, the DSP 106 in Fig. 2. The signal processing unit 126 performs various signal processes on the captured image before or after stabilization processing. The signal processing unit 126 transmits captured image data corresponding to the captured image before stabilization processing to the external terminal 12 via the I / O interface unit 128. The signal processing unit 126 supplies captured image data corresponding to the captured image after stabilization processing to the image cropping unit 127.

[0046] The image cropping unit 127 crops an image of a predetermined size from the captured image after stabilization processing based on the mesh information. The image cropping unit 127 transmits cropped image data corresponding to the cropped captured image (hereinafter referred to as a cropped image) to the external terminal 12 via the I / O interface unit 128.

[0047] The I / O interface unit 128 is connected to the external terminal 12 by wire or wirelessly, and communicates with the external terminal 12. When the I / O interface unit 128 is connected to the external terminal 12 by wire, it includes, for example, a commonly used external connection connector.

[0048] The external terminal 12 includes an I / O interface unit 151, a UI processing unit 152, a mesh creation unit 153, an offset adjustment unit 154, a detection unit 155, and a display operation unit 156. The I / O interface unit 151, the UI processing unit 152, the mesh creation unit 153, the offset adjustment unit 154, the detection unit 155, and the display operation unit 156 are connected to each other via a bus.

[0049] The I / O interface unit 151 is connected to the camera 11 by wire or wirelessly, and communicates with the camera 11. When the I / O interface unit 151 is connected to the camera 11 by wire, it includes, for example, a commonly used external connection connector.

[0050] The UI processing unit 152 executes processing related to the user interface of the external terminal 12. For example, the UI processing unit 152 controls the display of the display operation unit 156.

[0051] The mesh creation unit 153 receives parameter information from the camera 11 via the I / O interface unit 151. Based on the parameter information, the mesh creation unit 153 creates a mesh in the camera coordinate system so as to have a desired format for the camera 11. The mesh creation unit 153 generates mesh information regarding the created mesh and transmits it to the camera 11 via the I / O interface unit 151.

[0052] The offset adjustment unit 154 receives motion sensor information from the camera 11 via the I / O interface unit 151. Based on the motion sensor information, the offset adjustment unit 154 adjusts the offset value of the motion sensor 105. The offset adjustment unit 154 generates offset information indicating the offset value of the motion sensor 105 after adjustment, and transmits the offset information to the camera 11 via the I / O interface unit 151.

[0053] The detection unit 155 receives captured image data and cropped image data from the camera 11 via the I / O interface unit 151. The detection unit 155 detects feature points in the captured image data and cropped image data, and supplies information about the detected feature points to the UI processing unit 152. The detection unit 155 detects a stabilization center based on the captured image data and cropped image data, and detects the principal point of the camera 11 based on the detection result of the stabilization center. The detection unit 155 generates principal point information indicating the detection result of the principal point of the camera 11, and transmits it to the camera 11 via the I / O interface unit 151.

[0054] The display operation unit 156 is configured by, for example, a combination of a display device and an operation member, such as a touch panel or a monitor and a keyboard, etc. The display operation unit 156 is used, for example, to display various information and to input operation signals and various information to the camera 11 and the external terminal 12.

[0055] For example, under the control of the UI processing unit 152, the display operation unit 156 displays the captured image, the cropped image, the feature points and optical flow of the captured image and the cropped image, the measurement results of the motion sensor 105, and the like.

[0056] For example, the display operation unit 156 displays a setting screen under the control of the UI processing unit 152, and enables various pieces of information to be set, created, checked, etc. Information that can be set, created, checked, etc. on the setting screen includes, for example, internal parameters of the camera 11, mesh information of the camera coordinate system, setting information related to feature point detection, setting information related to stabilization processing, setting information related to stabilization center detection processing, setting information related to signal processing, and setting information related to shooting (for example, aperture, shutter speed, gain, AE (auto exposure), etc.). In addition, information that can be set, created, checked, etc. on the setting screen includes, for example, information on each part of the camera 11 and the external terminal 12.

[0057] The display operation unit 156 supplies input information relating to input operation signals and various types of information to each unit of the external terminal 12. The display operation unit 156 also transmits the input information to the camera 11 via the I / O interface unit 151.

[0058] 3 is an example and can be changed as appropriate. For example, the division of functions between the camera 11 and the external terminal 12 can be changed.

[0059] For example, as shown in FIG. 4, the mesh creating unit 153 and the offset adjusting unit 154 can be provided in the camera 11 instead of the external terminal 12.

[0060] For example, as shown in FIG. 5, it is possible to provide a mesh creating unit 153, an offset adjusting unit 154, and a detecting unit 155 in the camera 11 instead of in the external terminal 12.

[0061] For example, as shown in Fig. 6, it is possible to provide all of the functions of the external terminal 12 in Fig. 2 in the camera 11. In this case, it becomes possible to remove the external terminal 12 from the measurement system 1.

[0062] 3 to 5, for example, the mesh setting unit 122 and the mesh creating unit 153 can be integrated into one unit. For example, the offset setting unit 123 and the offset adjusting unit 154 can be integrated into one unit.

[0063] Furthermore, for example, in the configuration examples shown in FIGS. 2 to 4, the camera 11 may be provided with an operation unit and a display unit.

[0064] In the following, an example in which the camera 11 and the external terminal 12 are configured as shown in FIG. 3 will be described.

[0065] <Configuration example of rotation mechanism 13> Next, a configuration example of the rotation mechanism 13 in FIG. 1 will be described with reference to FIGS.

[0066] <First embodiment of the rotation mechanism 13> FIG. 7 shows a first embodiment of the rotation mechanism 13.

[0067] The rotation mechanism 13 includes a rotation jig 201, a rotation shaft 202, and an attachment jig 203. The rotation jig 201 and the attachment jig 203 are connected via the rotation shaft 202.

[0068] The rotation jig 201 is a jig for rotating the rotation shaft 202. For example, the rotation jig 201 is configured by a handle or the like, which will be described later.

[0069] The mounting jig 203 is a jig for mounting the camera 11 to the rotation mechanism 13 .

[0070] For example, when a user rotates a rotation axis 202 using a rotation jig 201, an attachment jig 203 connected to the rotation axis 202 and a camera 11 attached to the attachment jig 203 rotate.

[0071] <Second embodiment of the rotation mechanism 13> Fig. 8 shows a second embodiment of the rotation mechanism 13. In the figure, parts corresponding to those in the rotation mechanism 13 of Fig. 7 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0072] The rotation mechanism 13 of FIG. 8 has a configuration in which an angle jig 204 is added to the rotation mechanism 13 of FIG.

[0073] Angle jig 204 is provided between rotation jig 201 and mounting jig 203. Angle jig 204 is a jig that can change the angles in the pitch direction and yaw direction of mounting jig 203. By adjusting the angles in the pitch direction and yaw direction of angle jig 204, the angles in the pitch direction and yaw direction of camera 11 attached to mounting jig 203 can be adjusted.

[0074] <Third embodiment of the rotation mechanism 13> 9 and 10 show a third embodiment of the rotation mechanism 13. Fig. 9 is a plan view of the rotation mechanism 13, and Fig. 10 is a block diagram showing an example of the configuration of the rotation mechanism 13. In the figures, parts corresponding to those of the rotation mechanism 13 in Fig. 8 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0075] The rotation mechanism 13 includes a rotation shaft 202, a mounting jig 203, a mounting base 221, a handle 222, a bearing 223-1, and a bearing 223-2.

[0076] The rotating shaft 202 is fixed and rotatable by bearings 223-1 and 223-2 installed on the installation base 221. A handle 222 is attached to one end of the rotating shaft 202, and a mounting jig 203 is attached to the other end of the rotating shaft 202. The camera 11 is attached to the mounting jig 203.

[0077] By rotating the handle 222, the user can rotate the camera 11 in the roll direction around the rotation axis 202.

[0078] As shown by the dotted line in FIG. 10, an angle jig 204 can be installed between the rotation axis 202 and the mounting jig 203.

[0079] <Fourth embodiment of the rotation mechanism 13> 11 and 12 show a fourth embodiment of the rotation mechanism 13. Fig. 11 is a plan view of the rotation mechanism 13, and Fig. 12 is a block diagram showing an example of the configuration of the rotation mechanism 13. In the figures, parts corresponding to those of the rotation mechanism 13 in Figs. 9 and 10 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0080] The rotation mechanism 13 includes a mounting jig 203, a handle 222, bearings 223-1 to 223-4, a mounting base 241, a rotating shaft 242, gears 243-1 and 243-2, a timing belt 244, a rotating shaft 245, a cable 246, and an I / O interface unit 247.

[0081] The rotating shaft 242 is fixed and rotatable by bearings 223-1 and 223-2 installed on the installation base 221. A handle 222 is installed on one end of the rotating shaft 242. A gear 243-1 is installed near the center of the rotating shaft 242.

[0082] The rotating shaft 245 is fixed so as to be parallel to the rotating shaft 242 and is rotatable by bearings 223-3 and 223-4 installed on the installation base 221. An attachment jig 203 is installed on one end of the rotating shaft 245. A gear 243-2 is installed on the rotating shaft 245 at a position corresponding to the gear 243-1 of the rotating shaft 242.

[0083] The gear 243-1 and the gear 243-2 are connected by a timing belt 244. Therefore, by rotating the handle 222, the rotation shaft 242 rotates, and the rotation of the rotation shaft 242 is transmitted to the rotation shaft 245 by the gear 243-1, the timing belt 244, and the gear 243-2. ​​This causes the rotation shaft 245 to rotate, and the camera 11 attached to the attachment jig 203 rotates in the roll direction around the rotation shaft 245.

[0084] The rotation speed of the camera 11 can be changed by changing the gear ratio between the gear 243-1 and the gear 243-2.

[0085] A cable 246 passes through the inside of the rotation shaft 245 and connects the camera 11 to an I / O interface unit 247 .

[0086] 3 to 5, for example, is connected by wire or wirelessly to the external terminal 12, and communicates with the external terminal 12. When the I / O interface unit 247 is connected by wire to the external terminal 12, it includes, for example, a commonly used external connection connector.

[0087] On the other hand, in the embodiment of FIG. 6, the functions of the external terminal 12 are installed in the camera 11, and therefore the I / O interface unit 247 is omitted.

[0088] As shown by the dotted line in FIG. 12, an angle jig 204 can be installed between the rotation axis 245 and the mounting jig 203.

[0089] For example, a motor may be provided instead of the handle 222 to mechanically rotate the rotary shaft 202 or the rotary shafts 242 and 245. For example, a stepping motor may be used as the motor.

[0090] Furthermore, for example, without providing the rotation mechanism 13, it is possible for the user to manually rotate the camera 11 in the roll direction, or to manually rotate the camera 11 in the roll direction and set it at a predetermined rotation position (angle in the roll direction), as described below.

[0091] Furthermore, the number of bearings shown in FIGS. 9 to 12 is merely an example, and any number can be set.

[0092] <Relationship between the deviation of the center of the lens 103 and the rotation axis AR1 and the stabilization process>

[0093] Next, with reference to FIGS. 13 to 20, the relationship between the deviation between the center of the lens 103 of the camera 11 and the rotation axis AR1 of the rotation mechanism 13 and the stabilization process will be described.

[0094] 13 shows an example in which the center CL1 of the lens 103 of the camera 11 overlaps with the rotation axis AR1 of the rotation mechanism 13. In this case, the rotation axis AR1 passes through the center CL1 of the lens 103 and substantially overlaps with the optical axis AO1 of the lens 103.

[0095] FIG. 14 shows an example of the locus of feature points in a captured image when the measurement target 14 is photographed in a case where the center CL1 of the lens 103 and the rotation axis AR1 overlap.

[0096] 14A, the center of the camera coordinate system and the center of the image coordinate system (captured image) are substantially aligned with the rotation axis AR1. In this example, the measurement object 14 has a pattern of 3 rows x 3 columns of dots (feature points).

[0097] 14B shows an example of the trajectory of feature points in a captured image when the camera 11 is rotated in the roll direction around the rotation axis AR1 without executing stabilization processing. In this example, as the camera 11 rotates, the feature points move in a circle around the center of the captured image.

[0098] 14C shows an example of the trajectory of feature points in a captured image when the camera 11 is rotated in the roll direction around the rotation axis AR1 while the stabilization process is being performed. In this example, the feature points remain stationary and do not move even when the camera 11 rotates.

[0099] Therefore, in this case, the stabilization center can be accurately detected in the stabilization center detection process described later.

[0100] 15 shows an example in which the center CL1 of lens 103 of camera 11 is misaligned by a distance t from the rotation axis AR1 of rotation mechanism 13. In this case, a misalignment of a distance t also occurs between the rotation axis AR1 and the optical axis AO1 of lens 103.

[0101] FIG. 16 shows an example of the locus of feature points in a captured image when the measurement target 14 is photographed in a case where the center CL1 of the lens 103 and the rotation axis AR1 are shifted by a distance t.

[0102] As shown in A of FIG. 16, the center of the camera coordinate system and the center of the image coordinate system (captured image) are shifted from the rotation axis AR1 by a distance t.

[0103] 16B shows an example of the trajectory of a feature point in a captured image when the camera 11 is rotated in the roll direction around the rotation axis AR1 without executing stabilization processing. In this case, the center of the image coordinate system rotates in accordance with the rotation of the camera 11. Therefore, in the captured image, the feature point moves by an amount that is the sum of the rotation component accompanying the rotation of the camera 11 and the rotation component of the center of the image coordinate system.

[0104] 16C shows an example of the trajectory of feature points in a captured image when the camera 11 is rotated in the roll direction around the rotation axis AR1 while performing stabilization processing. In this case, the feature points move in a circular shape in accordance with the rotation of the center of the image coordinate system.

[0105] Therefore, in this case, the stabilization center cannot be accurately detected in the stabilization center detection process described later.

[0106] Therefore, it is desirable to adjust at least one of the positions of the camera 11 and the mounting jig 203 so that the center CL1 of the lens 103 and the rotation axis AR1 coincide with each other. Alternatively, it is desirable to use a mounting jig 203 such that the center CL1 of the lens 103 and the rotation axis AR1 coincide with each other when the camera 11 is mounted.

[0107] Alternatively, for example, as shown in FIG. 17, an XY stage 251 may be used as the mounting jig 203.

[0108] The XY stage 251 is a member that can move the stage on which the camera 11 is installed in the horizontal and vertical directions. As a result, even if there is a misalignment between the center CL1 of the lens 103 and the rotation axis AR1, for example, as shown in A of Fig. 17, by moving the XY stage 251, the center CL1 of the lens 103 can be aligned with the rotation axis AR1, as shown in B of Fig. 17.

[0109] Alternatively, for example, the amount of deviation between the center of the camera coordinate system and the center of the image coordinate system and the rotation axis AR1 may be corrected.

[0110] Here, an example of a method for calculating the amount of deviation between the center of the camera coordinate system and the center of the image coordinate system and the rotation axis AR1 will be described with reference to FIGS.

[0111] 18, the center of rotation when the camera 11 rotates by rotating the rotation mechanism 13 is defined as CR1. The center of rotation CR1 coincides with the rotation axis AR1 of the rotation mechanism 13. Hereinafter, the height (vertical distance) between the center of rotation CR1 and the center CL1 of the lens 103 is defined as H LENS Let the width (horizontal distance) be W LENS Hereafter, the height of the camera 11 is H CAM and the width is W CAMLet's say.

[0112] 19, the amount of deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR1 is defined as t. Hereinafter, the amount of deviation in the height (vertical) direction of the amount of deviation t will be referred to as t h The horizontal deviation is t w Hereinafter, the point where the measurement object 14 intersects with the rotation axis AR1 is referred to as PC1. Hereinafter, the amount of deviation between the center of the image coordinate system and the rotation axis AR1 is referred to as t f Hereafter, the amount of deviation t f The amount of deviation in the height (vertical) direction is t fh The horizontal deviation is t fw Let's say.

[0113] 20A shows the relationship between the amount of deviation t in the camera coordinate system and the distance L between the camera 11 and the measurement target 14. FIG. 20B shows the relationship between the amount of deviation t in the image coordinate system and the distance L between the camera 11 and the measurement target 14. f and the focal length f of the lens 103.

[0114] Here, the triangle A in FIG. 20 and the triangle B in FIG. 20 are similar. Also, the amount of deviation t v and the amount of deviation t h is expressed by the following equations (1) and (2).

[0115] t h =H LENS ···(1) t w =W LENS ···(2)

[0116] Therefore, the displacement amount t in the image coordinate system fv and the amount of deviation t fh is expressed by the following equations (3) and (4).

[0117] t fh =f×t h / L=f×H LENS / L (3) t fw =f×t w / L=f×W LENS / L (4)

[0118] Here, for example, H LENS = 1 mm, W LENS = 1 mm, L = 2 m, and the focal length of the camera 11 converted into pixels is f = 1000 pixels, the amount of deviation t fv and the amount of deviation t fh is 0.5 pixels as shown in the following equations (5) and (6).

[0119] t fh =1000×1 / 2000=0.5 (5) t fw =1000×1 / 2000=0.5 (6)

[0120] For example, the detection unit 155 of the external terminal 12 may LENS , W LENS , distance L, and focal length f are acquired from the outside, and the displacement amount t fv and the amount of deviation t fh After detecting the stabilization center in the process described later, the detection unit 155 calculates the amount of deviation t fv and the amount of deviation t fh This corrects the error in the stabilization center caused by the deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR1 of the rotation mechanism 13, improving the detection accuracy of the stabilization center.

[0121] For example, the distance between the camera 11 and the measurement object 14 may be set to an infinity equivalent. fv and the amount of deviation t fh becomes essentially 0, and correction of the detected value of the stabilization center becomes unnecessary.

[0122] <Configuration example of angle jig 204> Next, a configuration example of the angle jig 204 will be described with reference to FIGS.

[0123] The examples of FIGS. 21 and 22 differ from the example of FIG. 13 in that an angle jig 204 is provided between the rotation axis 202 and the mounting jig 203.

[0124] The angle jig 204 is a jig for tilting the optical axis AO1 of the lens 103 of the camera 11 by an angle θ with respect to the rotation axis AR1 of the rotation mechanism 13. By using the angle jig 204, for example, the camera 11 can be installed so that the orientation of the optical axis AO1 (the Z axis of the camera coordinate system) with respect to the rotation axis AR1 is at an angle θ.

[0125] This allows the amount of change in the motion sensor 105 (angular velocity and acceleration of the camera 11) to be increased when the camera 11 is rotated by the rotation mechanism 13, improving the accuracy of detecting the center of stabilization.

[0126] In addition, the angle jig 204 may be configured so that the orientation (angle θ) of the camera 11 can be changed, or the orientation of the camera 11 may be fixed.

[0127] Here, it is desirable to set the angle θ to a value sufficiently smaller than 1 / 2 (Φ / 2) of the viewing angle Φ of the lens 103. If the angle θ is set to Φ / 2 or more, the measurement object 14 may go outside the image coordinate system when the camera 11 rotates, making it impossible to detect feature points of the measurement object 14 and the stabilization center.

[0128] As shown in FIG. 21, it is desirable to install the mounting jig 203 and the angle jig 204 so that the rotation axis AR1 of the rotation mechanism 13 and the center CL1 of the lens 103 coincide with each other.

[0129] On the other hand, as shown in FIG. 22, if the rotation axis AR1 of the rotation mechanism 13 and the center CL1 of the lens 103 are misaligned, the amount of misalignment can be corrected in the process of detecting the stabilization center, as described above.

[0130] Also, for example, as shown in FIG. 23, by using an XY stage 251 as a mounting jig 203, the position of the camera 11 may be adjusted so that the center CL1 of the lens 103 coincides with the rotation axis AR1.

[0131] <Example of measurement object 14> Next, examples of the measurement object 14 will be described with reference to FIGS.

[0132] FIG. 24 shows an example in which the measurement object 14 is a board, which is a plate-like member that shows a pattern including a single feature point.

[0133] FIG. 24A shows an example in which a board with a circular mark arranged at the center is used as the measurement object 14.

[0134] FIG. 24B shows an example in which a board showing a set of gratings is used as the measurement object 14.

[0135] FIG. 25 shows an example in which the measurement object 14 is a board, which is a plate-like member showing a pattern including a plurality of feature points.

[0136] FIG. 25A shows an example in which a board on which a plurality of circular marks are arranged in a grid pattern is used as the measurement object 14.

[0137] FIG. 25B shows an example in which a board showing a lattice pattern is used as the measurement object 14.

[0138] When a board showing a predetermined pattern is used as the measurement object 14, it is assumed that the measurement object 14 will be installed indoors. Therefore, it is desirable to set the distance L between the camera 11 and the measurement object 14 to be several centimeters to several tens of meters.

[0139] Note that grid patterns such as those shown in Figures 24B and 25B may appear distorted when viewed from an oblique angle, which may reduce the accuracy of feature point detection. Therefore, it is preferable to use a board with circular marks such as those shown in Figures 24A and 25A.

[0140] FIG. 26 shows an example in which an external object is used as the measurement object 14.

[0141] FIG. 26A shows an example in which an outdoor house or building is used as the measurement object 14.

[0142] FIG. 26B shows an example in which an outdoor sign or a utility pole is used as the measurement object 14.

[0143] 26, any object whose feature points can be detected and whose feature points are stationary can be used as the external object used for the measurement object 14. For example, fixed objects such as trees, bridges, mountains, and rivers, or stationary stars such as the North Star can be used as the measurement object 14.

[0144] When an external object is used as the measurement object 14, it is desirable to set the distance L between the camera 11 and the measurement object 14 to several tens of meters or more, or to a distance equivalent to infinity.

[0145] <Calibration process> Next, the calibration process executed by the measurement system 1 will be described with reference to the flowchart of FIG.

[0146] First, before the calibration process, an environment for detecting the stabilization center is constructed as shown in Fig. 1. That is, a camera 11 is attached to a rotation mechanism 13, and a measurement target 14 is placed facing the camera 11.

[0147] In the following, an example will be described in which the size of the image sensor 104 is 4000 pixels vertically by 4000 pixels horizontally, and the lens 103 is configured as a fisheye lens with a viewing angle of 180° and a focal length in pixels of 1000. Also, it is assumed that the camera 11 and the rotation mechanism 13 are installed so that the center CL1 of the lens 103 and the rotation axis AR1 of the rotation mechanism 13 overlap.

[0148] In step S1, the measurement system 1 captures and displays an image of the measurement target 14.

[0149] Specifically, the image sensor 104 of the camera 11 captures an image of the measurement object 14 via the lens 103, and supplies the obtained captured image data to the signal processing unit 126. The signal processing unit 126 performs various signal processing on the captured image, and transmits the captured image data after the signal processing to the external terminal 12 via the I / O interface unit 128.

[0150] In response to this, the I / O interface unit 151 of the external terminal 12 receives the captured image data from the external terminal 12. The I / O interface unit 151 supplies the captured image data to each unit of the external terminal 12. The display operation unit 156 displays the captured image in which the measurement object 14 is captured, based on the captured image data. As a result, for example, as shown in FIG. 28 , the captured image in which the measurement object 14 is captured is displayed on the display operation unit 156.

[0151] In step S2, the external terminal 12 acquires the internal parameters of the camera 11. For example, the user inputs the internal parameters of the camera 11 via the display operation unit 156 of the external terminal 12.

[0152] The internal parameters of the camera 11 include the principal point (Cx, Cy), focal length (fx, fy), and correction coefficients k1 to k4, p1, and p2 of the camera 11. These internal parameters are used for distortion correction and stabilization processing of the lens 103, etc.

[0153] For example, distortion correction of the lens 103 is performed using the following equations (7) to (12): The coordinates (Xc, Yc, Zc) are coordinates in the camera coordinate system of the camera 11, and the coordinates (u, v) are coordinates in the image coordinate system of the captured image.

[0154]

number

[0155] When the lens 103 is a fisheye lens, distortion correction of the lens 103 is performed using, for example, the following equations (13) to (20).

[0156]

number

[0157] For example, the focal lengths (fx, fy) and the correction coefficients k1 to k4, p1, and p2 are specified or guaranteed by the lens manufacturer, or parameters measured using a lens parameter measuring device are used.

[0158] For example, the coordinates of the center of the captured image are provisionally set as the principal point (Cx, Cy).

[0159] The display operation unit 156 generates input information including the input internal parameters and supplies it to each unit of the external terminal 12. The display operation unit 156 also transmits the input information to the camera 11 via the I / O interface unit 151.

[0160] In response to this, the parameter setting unit 121 of the camera 11 receives the input information via the I / O interface unit 128. The parameter setting unit 121 sets internal parameters of the camera 11 based on the input information. The parameter setting unit 121 generates parameter information related to the set internal parameters and supplies the parameter information to the stabilization processing unit 124.

[0161] In step S3, the measurement system 1 creates and sets a mesh of the camera coordinate system.

[0162] Specifically, the mesh creating unit 153 of the external terminal 12 creates a mesh in the camera coordinate system based on the internal parameters of the camera 11 acquired in the processing of step S2.

[0163] As a result, for example, in comparison with the mesh in the normal camera coordinate system as shown in A of FIG. 29, a mesh in the camera coordinate system that reflects the distortion of the lens 103 as shown in B of FIG. 29 is created.

[0164] Any method can be applied to create the mesh of the camera coordinate system. For example, the methods described in Patent Document 1, Japanese Patent Laid-Open No. 2013-134088, or International Publication No. 2020-039747 can be applied.

[0165] Alternatively, for example, a plurality of types of meshes may be prepared in advance, and an appropriate mesh may be selected from among them based on the internal parameters of the camera 11.

[0166] The mesh creating unit 153 generates mesh information relating to the created mesh and transmits it to the camera 11 via the I / O interface unit 151 .

[0167] In response to this, the mesh setting unit 122 of the camera 11 receives the mesh information from the external terminal 12 via the I / O interface unit 128. The mesh setting unit 122 sets a mesh in the camera coordinate system based on the mesh information. The mesh setting unit 122 generates mesh information related to the set mesh and supplies it to the image cropping unit 127.

[0168] In step S4, the display operation unit 156 of the external terminal 12 displays the center of the captured image under the control of the UI processing unit 152. For example, as shown in A of Fig. 30, a cross mark, which is information indicating the position of the center of the captured image, is superimposed on the captured image displayed on the display operation unit 156.

[0169] In contrast, for example, as shown in B of Figure 30, if the center of the captured image and the center of the measurement object 14 are misaligned, the user adjusts the position of the measurement object 14 as necessary so that the center of the captured image overlaps the center of the measurement object 14.

[0170] In step S5, the external terminal 12 detects and displays feature points of the captured image.

[0171] Specifically, the detection unit 155 of the external terminal 12 detects feature points of the captured image.

[0172] Any method can be used to detect feature points, such as Moravec's corner detection algorithm, Harris, Stephens, and Plessey's corner detection algorithm, multi-scale Harris operator, Shi and Tomasi's method, curvature-based method, Laplacian-Gaussian and derivative method, Wang and Brady's method, Trajkovic and Hedley's method, and FAST feature detection.

[0173] The detection unit 155 supplies information about the detected feature points to the UI processing unit 152. Under the control of the UI processing unit 152, the display operation unit 156 superimposes information indicating the positions of the feature points on the captured image being displayed.

[0174] As a result, for example, as shown in Figure 31, the center of the measurement object 14 in the captured image is detected as feature point PF1, and information indicating the position of the detected feature point PF1 is superimposed on the captured image being displayed on the display operation unit 156.

[0175] For example, when multiple feature points are detected and displayed, the user may use the display operation unit 156 to select one feature point to be used for detecting the stabilization center, and may not display the other feature points.

[0176] Furthermore, for example, if a feature point is not detected or the feature point detection result is unstable, the user may use the display operation unit 156 to change the feature point detection method or detection parameters.

[0177] Next, the user uses the rotation mechanism 13 to rotate the camera 11 in the roll direction.

[0178] 32A, for example, the display operation unit 156 superimposes the trajectory of the feature point PF1 on the captured image under the control of the UI processing unit 152. Then, the user confirms that the feature point PF1 does not disappear and continues to be displayed on the display operation unit 156 even when the camera 11 is rotated. In other words, the user confirms that the feature point PF1 does not go outside the viewing angle Φ1 of the camera 11 even when the camera 11 is rotated.

[0179] If rotating the camera 11 causes the feature point PF1 to disappear from the display operation unit 156, the user adjusts the position of the measurement object 14 so that the feature point PF1 does not go outside the viewing angle Φ1 of the camera 11.

[0180] Furthermore, for example, the user adjusts the position of the measurement object 14 as needed so that the center of the circle indicating the locus of the feature point PF1 substantially overlaps with the center of the captured image, as shown in B of FIG.

[0181] In step S6, the camera 11 starts the stabilization process.

[0182] Specifically, for example, the user uses an operation unit of the camera 11 (not shown) to perform an operation to set the image stabilization function to ON.

[0183] In response to this, the UI processing unit 125 turns on the image stabilization function and notifies each unit of the camera 11 that the image stabilization function has been turned on.

[0184] The stabilization processing unit 124 performs stabilization processing. For example, the stabilization processing unit 124 performs stabilization processing by transforming the coordinates (u, v) of the captured image into coordinates (u', v') using a rotation matrix R and a projection transformation matrix K, as shown in FIG. 33 and the following equation (21).

[0185]

number

[0186] The rotation matrix R and the projection transformation matrix K are expressed by the following equations (22) to (24).

[0187]

number

[0188] θ pitch , θ yaw , and θ roll represent the rotation angle of the camera 11 in the pitch direction, the rotation angle in the yaw direction, and the rotation angle in the roll direction, respectively.

[0189] As the posture information of the camera 11, a rotation vector (θ pitch ,θ yaw ,θ roll ), for example, quaternion, rotation matrix SO(3), Euler angles, etc. can also be used.

[0190] The stabilization processing unit 124 supplies the photographed image data after the stabilization processing to the image cropping unit 127 .

[0191] The image cropping unit 127 crops an image of a predetermined size from the captured image based on the mesh information supplied from the mesh setting unit 122. For example, the image cropping unit 127 crops a cropped image of 1920 pixels vertically by 1080 pixels horizontally from a captured image of 4000 pixels vertically by 4000 pixels horizontally. For example, if the viewing angle of the captured image is 180°, the viewing angle of the cropped image can be set to 90°. Note that the viewing angle of the cropped image can also be set to an angle other than 90°.

[0192] The image cropping unit 127 transmits cropped image data corresponding to the cropped image cropped from the captured image to the external terminal 12 via the I / O interface unit 128.

[0193] In response to this, the I / O interface unit 128 of the external terminal 12 receives the cropped image data and supplies it to each unit of the external terminal 12. The detection unit 155 detects feature points of the cropped image based on the cropped image data, and supplies information on the detected feature points to the UI processing unit 152.

[0194] The display operation unit 156 displays the cut-out image based on the cut-out image data under the control of the UI processing unit 152. Furthermore, the display operation unit 156 superimposes information indicating the positions of the cross marks and feature points on the cut-out image under the control of the UI processing unit 152.

[0195] As a result, for example, as shown in FIG. 34, a cut-out image including the measurement target 14 and having information indicating the position of the cross mark and the characteristic point PF1 superimposed thereon is displayed on the display operation unit 156.

[0196] In step S7, the external terminal 12 starts tracking the feature points.

[0197] Specifically, for example, the user uses the display operation unit 156 to perform an operation to set the optical flow function to ON.

[0198] In response to this, the UI processing unit 152 turns on the optical flow function and notifies each unit of the external terminal 12 that the optical flow function has been turned on. The detection unit 155 starts tracking feature points in the cropped image and supplies information on the tracking results of the feature points to the UI processing unit 152.

[0199] Next, the user uses the rotation mechanism 13 to rotate the camera 11 in the roll direction.

[0200] 35, for example, the display operation unit 156 superimposes information indicating the trajectory of the feature point PF1 on the cut-out image under the control of the UI processing unit 152. Then, the user confirms that the feature point PF1 does not disappear and continues to be displayed on the display operation unit 156 even when the camera 11 is rotated. In other words, the user confirms that the feature point PF1 does not protrude from the cut-out image even when the camera 11 is rotated.

[0201] For example, as shown in A of Figure 36, if rotating the camera 11 causes the feature point PF1 to extend beyond the cut-out image, the user adjusts the position of the object to be measured 14 so that the feature point PF1 does not extend beyond the cut-out image.

[0202] Furthermore, the user adjusts the position of the measurement object 14 so that the center of the locus of the feature point PF1 substantially overlaps with the center of the cropped image, as shown in B of FIG.

[0203] In step S8, the offset setting unit 123 of the camera 11 determines whether or not the offset of the motion sensor 105 has been corrected. If it is determined that the offset of the motion sensor 105 has not been corrected, the process proceeds to step S9.

[0204] In step S9, the measurement system 1 corrects the offset of the motion sensor 105.

[0205] For example, the user keeps the camera 11 stationary without moving it for a predetermined time (e.g., 10 seconds) or more. The motion sensor 105 measures the angular velocity of the motion sensor 105 during the stationary period, i.e., the zero point of the angular velocity of the motion sensor 105. The angular velocity of the motion sensor 105 is represented by, for example, angular velocities Gx, Gy, and Gz on the X-axis, Y-axis, and Z-axis of the camera coordinate system.

[0206] Figure 37 shows examples of angular velocity measurements of the motion sensor 105 during a stationary period. Figure 37A shows an example of the measured value of angular velocity Gx. Figure 37B shows an example of the output value of angular velocity Gy. Figure 37C shows an example of the measured value of angular velocity Gz.

[0207] In this example, the zero point of the angular velocity Gx is -0.1987 dps, the zero point of the angular velocity Gy is 0.292656 dps, and the zero point of the angular velocity Gz is -0.34035 dps.

[0208] The offset setting unit 123 generates motion sensor information including the measurement results of the angular velocity of the motion sensor 105 during the stationary period, and transmits the information to the external terminal 12 via the I / O interface unit 128 .

[0209] In response to this, the offset adjustment unit 154 of the external terminal 12 receives the motion sensor information from the camera 11 via the I / O interface unit 151.

[0210] The offset adjustment unit 154 calculates the averages of the measured values ​​of the angular velocities Gx, Gy, and Gz during the stationary period as the zero points of the angular velocities Gx, Gy, and Gz, respectively. The offset adjustment unit 154 calculates offset values ​​for correcting the zero point values ​​of the angular velocities Gx, Gy, and Gz of the motion sensor 105 to 0. The offset adjustment unit 154 transmits offset information indicating the calculated offset values ​​to the camera 11 via the I / O interface unit 151.

[0211] In response to this, the offset setting unit 123 of the camera 11 receives the offset information from the external terminal 12 via the I / O interface unit 128. The offset setting unit 123 sets an offset value for the angular velocity of the motion sensor 105 based on the offset information.

[0212] Although a detailed description will be omitted, the offset values ​​of the acceleration of the motion sensor 105 on the X-axis, Y-axis, and Z-axis of the camera coordinate system are set in the same manner as the offset values ​​of the angular velocity.

[0213] Then, the process proceeds to step S10.

[0214] On the other hand, if it is determined in step S8 that the offset of the motion sensor 105 has been corrected, the process of step S9 is skipped and the process proceeds to step S10.

[0215] In step S10, the measurement system 1 executes a process for detecting the stabilization center.

[0216] For example, the detection unit 155 sets the position of the feature point PF1 in the image coordinate system of the cropped image after the camera 11 is kept stationary for a predetermined time (for example, 10 seconds) as the initial value of the stabilization center.

[0217] Next, the user rotates camera 11 in the roll direction a predetermined number of times (e.g., five times) using rotation mechanism 13. While camera 11 is rotating, detection unit 155 detects the position of feature point PF1 in the image coordinate system of the cropped image for each frame obtained by camera 11 capturing images in multiple postures with different angles in the roll direction. Detection unit 155 detects the center and radius of the trajectory of feature point PF1 for each rotation as the stabilization center and the variation in the position of feature point PF1, respectively.

[0218] Any method can be used to detect the center and radius of the locus of the feature points PF1. For example, the detection unit 155 uses a Hough transform to detect a circle that is the locus of the feature points PF1, and then detects the center and radius of the detected circle.

[0219] The following shows an example of the initial value of the stabilization center, and the detection results of the variance in the positions of the stabilization center and feature point PF1 for the first to fifth rotations.

[0220] Initial value of stabilization center = (910.94,518.66) Stabilization center of first rotation = (979.11,528.67) Stabilization center of the second rotation = (979.25, 529.94) Stabilization center of the third rotation = (978.34, 530.68) Stabilization center of the fourth rotation = (978.28, 531.98) Stabilization center of the 5th rotation = (978.99,532.83) Variation of feature point positions in the first rotation = (±68.2, ±67.4) Variation of feature point positions in the second rotation = (±67.3, ±67.2) Variation of feature point positions in the third rotation = (±67.4, ±67.1) Variation of feature point position for the fourth rotation = (±67.5, ±67.3) Variation of feature point position at 5th rotation = (±66.7, ±67.3)

[0221] At this time, if there is a misalignment between the center of the camera coordinate system of the camera 11 and the rotation axis AR1 of the rotation mechanism 13, the detection unit 155 may correct the detection result of the stabilization center, for example, by the method described above with reference to Figures 18 to 20.

[0222] The detection unit 155 calculates the average of the stabilization centers and the average of the variations in the position of the feature point PF1. In the above example, the average of the stabilization centers and the average of the variations in the position of the feature point PF1 are as follows:

[0223] Stabilization center average = (978.80, 530.82) Average feature point position = (±67.4, ±67.3)

[0224] At this time, if there is a misalignment between the center of the camera coordinate system of the camera 11 and the rotation axis AR1 of the rotation mechanism 13, the detection unit 155 may correct the average of the stabilization centers, for example, by the method described above with reference to Figures 18 to 20.

[0225] The detection unit 155 calculates the difference between the initial value of the stabilization center and the average of the stabilization centers as the amount of deviation of the stabilization center. In the above example, the amount of deviation of the stabilization center is as follows:

[0226] Displacement of the stabilization center =(910.94,518.66)-(978.80,530.82) =(-67.85,-12.15) ≒(-68,-12)

[0227] At this time, if there is a deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR1 of the rotation mechanism 13, the detection unit 155 may correct the deviation of the stabilization center, for example, by the method described above with reference to Figures 18 to 20.

[0228] The detection unit 155 calculates the principal point of the camera 11 by subtracting the amount of deviation of the stabilization center from the currently set principal point of the camera 11 .

[0229] For example, in this case, the principal point of the camera 11 is temporarily set to (2000, 2000) in step S2. Therefore, the principal point of the camera 11 is calculated as follows.

[0230] Principal point of camera 11 = (2000, 2000) - (-68, -12) =(2068,2012)

[0231] The coordinates of the principal point of the camera 11 in this formula are the coordinates obtained by converting the stabilization center in the image coordinate system of the extracted image into the stabilization center in the image coordinate system of the captured image.

[0232] In step S11, the detection unit 155 determines whether or not to end the stabilization center detection process. If it is determined that the stabilization center detection process should be continued, the process proceeds to step S12.

[0233] In step S12, the measurement system 1 updates the principal point of the camera 11 used in the stabilization process.

[0234] Specifically, the detection unit 155 supplies principal point information indicating the principal point of the camera 11 calculated in the processing of step S10 to the mesh creation unit 153, and also transmits it to the camera 11 via the I / O interface unit 151.

[0235] In response to this, the parameter setting unit 121 of the camera 11 receives the principal point information from the external terminal 12 via the I / O interface unit 128. The parameter setting unit 121 updates the currently set principal point of the camera 11 to the value indicated in the principal point information. The parameter setting unit 121 supplies parameter information relating to internal parameters including the updated principal point of the camera 11 to the stabilization processing unit 124.

[0236] Thereafter, the process returns to step S3, and steps S3 to S12 are repeatedly executed until it is determined in step S11 that the stabilization center detection process is to be terminated. As a result, the stabilization center detection process is repeatedly executed while the principal point of the camera 11 used for the stabilization process in the next detection process is updated based on the stabilization center detection result.

[0237] On the other hand, if it is determined in step S11 that the stabilization center detection process is to be ended, the process proceeds to step S13. For example, the detection unit 155 determines that the stabilization center detection process is to be ended when the stabilization center detection process has been executed a predetermined number of times (for example, five times).

[0238] In step S13, the measurement system 1 corrects the principal point of the camera 11.

[0239] For example, the detection unit 155 of the external terminal 12 performs the stabilization center detection process multiple times, and determines the principal point of the camera 11 to be the principal point of the camera 11 that was used in the stabilization process in the detection process that resulted in the smallest average variation in the position of the feature point PF1.

[0240] For example, if the detection results after five detection processes are as follows, the average variation in the position of feature point PF1 in the third detection process is the smallest. Therefore, (1933, 1997), which is the principal point of camera 11 used in the third detection process, is determined to be the principal point of camera 11.

[0241] Principal point of camera 11 in the first detection process = (2000, 2000) Principal point of camera 11 in the second detection process = (1932,1990) Principal point of camera 11 in the third detection process = (1933, 1997) Principal point of camera 11 in the fourth detection process = (1933, 1996) Principal point of camera 11 in the fifth detection process = (1933,1997) Variation in feature point positions for the first detection process = (±67.4, ±67.3) Variation in feature point positions for the second detection process = (±5.5, ±5.5) Variation in feature point positions for the third detection process = (±0.85, ±0.85) Variation in feature point positions for the fourth detection process = (±0.91, ±0.90) Variation in feature point positions for the fifth detection process = (±0.85, ±0.86)

[0242] The detection unit 155 supplies principal point information indicating the set principal point of the camera 11 to the mesh creation unit 153 and also transmits it to the camera 11 via the I / O interface unit 151 .

[0243] In response to this, the parameter setting unit 121 of the camera 11 receives the principal point information from the external terminal 12 via the I / O interface unit 128. The parameter setting unit 121 corrects the currently set principal point of the camera 11 to the value indicated in the principal point information. The parameter setting unit 121 supplies parameter information relating to internal parameters including the corrected principal point of the camera 11 to the stabilization processing unit 124.

[0244] In step S14, the measurement system 1 corrects the camera coordinate system mesh.

[0245] Specifically, the mesh creation unit 153 of the external terminal 12 creates a mesh in the camera coordinate system based on the internal parameters including the principal point of the camera 11 corrected in the processing of step S13. The mesh creation unit 153 generates mesh information regarding the created mesh in the camera coordinate system and transmits it to the camera 11 via the I / O interface unit 151.

[0246] The mesh setting unit 122 of the camera 11 receives mesh information from the external terminal 12 via the I / O interface unit 128. The mesh setting unit 122 corrects the mesh in the camera coordinate system based on the received information. The mesh setting unit 122 supplies the mesh information on the corrected mesh to the image cropping unit 127.

[0247] Then, the calibration process ends.

[0248] After the calibration process is completed, the user checks the operation of the image stabilization function. Specifically, the user turns on / off the power of the camera 11, sets the image stabilization function to on, and then rotates the rotation mechanism 13 to check that the feature points of the measurement object 14 are stationary and do not move.

[0249] As described above, even if the attachment error or mounting error of the lens 103, image sensor 104, motion sensor 105, etc. is unknown, the stabilization center can be simply and easily detected for each individual camera 11 without using a highly flat checkerboard or the like. Also, the stabilization center can be detected in a state where the stabilization process is performed using the motion sensor 105 actually provided in the camera 11. Furthermore, the principal point of the camera 11 used in the stabilization process can be set based on the detected stabilization center.

[0250] This improves the accuracy of image stabilization for camera 11. Furthermore, even if mounting errors or assembly errors occur during the assembly process of camera 11, or if distortion occurs in the lens or housing while camera 11 is in use, the central axis of the stabilization process can be easily corrected.

[0251] <<2. Modifications>> Hereinafter, modifications of the above-described embodiment of the present technology will be described.

[0252] <Modification regarding the method of detecting the stabilization center> For example, while the user manually rotates the camera 11 in the roll direction without using the rotation mechanism 13, the detection unit 155 can detect the stabilization center using the method described above with reference to FIG.

[0253] In this case, since it is difficult to overlap the center of lens 103 with the rotation axis of camera 11, for example, the distance between camera 11 and measurement object 14 may be set to an infinity equivalent and camera 11 may be rotated. Furthermore, the trajectory of feature points in the cropped image when camera 11 is rotated does not necessarily have to be circular and may be, for example, elliptical or irregular. Then, just as in the case where the trajectory of feature points is circular, the center of the trajectory of feature points may be detected as the stabilization center.

[0254] In this case, as in the method described above, the cropped image, feature points, trajectories of the feature points, and the like are displayed on the display operation unit 158 ​​of the external terminal 12. Therefore, the user can manually rotate the camera 11 in the roll direction and detect the stabilization center while checking the captured image, the state of the feature points, and the like.

[0255] Furthermore, since the rotation mechanism 13 is no longer necessary, the stabilization center can be detected more easily.

[0256] Furthermore, for example, the camera 11 may be rotated in the roll direction with the image stabilization function set to on, and the object to be measured 14 may be photographed by the camera 11 while the camera 11 is installed in a number of positions with different angles in the roll direction, and the stabilization center may be detected based on the positions of the feature points of the cropped images obtained in each position.

[0257] For example, first, the user places the camera 11 on a stand or the like in a normal orientation, with the camera 11 facing the measurement object 14 and with the image stabilization function turned on. That is, the camera 11 is placed on a stand or the like with its bottom facing downwards. The camera 11 captures an image of the measurement object 14 in this first orientation. As shown in A of FIG. 38 , the detection unit 155 of the external terminal 12 detects a feature point PF1a of the measurement object 14 in an image cropped from the captured image captured in the first orientation.

[0258] Next, the user positions the camera 11 relative to the measurement object 14, rotates the camera 11 counterclockwise by 90 degrees in the roll direction by hand with the image stabilization function set to on, and places the camera on a stand or the like. That is, the camera 11 is placed on a stand or the like with its left side facing down. The camera 11 captures an image of the measurement object 14 in this second orientation. As shown in B of FIG. 38 , the detection unit 155 of the external terminal 12 detects the feature point PF1b of the measurement object 14 in the cropped image cropped from the captured image captured in the second orientation.

[0259] Next, the user points the camera 11 toward the measurement object 14, rotates the camera 11 counterclockwise by 90 degrees in the roll direction by hand with the image stabilization function set to on, and places the camera on a stand or the like. That is, the camera 11 is placed on a stand or the like with the top surface facing downwards (upside down). The camera 11 captures an image of the measurement object 14 in this third orientation. As shown in C of FIG. 38 , the detection unit 155 of the external terminal 12 detects the feature point PF1c of the measurement object 14 in the cropped image cropped from the captured image captured in the third orientation.

[0260] Next, the user positions the camera 11 relative to the measurement object 14, rotates the camera 11 counterclockwise by 90 degrees in the roll direction by hand with the image stabilization function set to on, and places the camera on a stand or the like. That is, the camera 11 is placed on a stand or the like with its right side facing down. The camera 11 captures an image of the measurement object 14 in this fourth orientation. The detection unit 155 of the external terminal 12 detects a feature point PF1d of the measurement object 14 in a cropped image cropped from the captured image captured in the fourth orientation, as shown in D of FIG.

[0261] Then, the detection unit 155 of the external terminal 12 detects the centers of the feature points PF1a to PF1d as the stabilization center in the image coordinate system of the cropped image, as shown in Fig. 39. For example, the detection unit 155 detects the intersection of a line connecting the feature points PF1a and PF1c with a line connecting the feature points PF1b and PF1d as the stabilization center.

[0262] In this case, as in the method described above, the cropped image, feature points, trajectories of the feature points, and the like are displayed on the display operation unit 158 ​​of the external terminal 12. Therefore, while checking the captured image and the state of the feature points, the user can manually rotate the camera 11 in the roll direction and detect the stabilization center while placing it in a predetermined posture.

[0263] Furthermore, since the rotation mechanism 13 is no longer necessary, the stabilization center can be detected more easily.

[0264] If it is difficult to align the center of lens 103 with the rotation axis of camera 11, the distance between camera 11 and measurement object 14 may be set to an amount equivalent to infinity, and camera 11 may be rotated. Alternatively, the detection value of the center of stabilization may be corrected by the amounts of deviation tfv and tfh between the center of the image coordinate system and the rotation axis, which are caused by the deviation between the center of lens 103 and the rotation axis of camera 11, using the method described above with reference to Figures 18 to 20.

[0265] Furthermore, the orientations at which the measurement object 14 is photographed are not necessarily limited to the above four orientations, and any three or more orientations with different roll direction angles can be adopted. However, it is desirable to leave as much space as possible between the roll direction angles of each orientation. The centers of the feature points detected in each of the multiple clipped images clipped from the multiple captured images photographed in each orientation are detected as the stabilization center. For example, the center of gravity of the feature points is used as the center of the feature points.

[0266] Furthermore, for example, in this method, the camera 11 may be rotated clockwise in the roll direction.

[0267] <Other variations> 27, the stabilization center detection process may be terminated when the average of the positional variation of feature point PF1 becomes equal to or less than a predetermined threshold. That is, the stabilization center detection process may be repeated until the average of the positional variation of feature point PF1 becomes equal to or less than the predetermined threshold. Then, the principal point of camera 11 used in the detection process in which the average of the positional variation of feature point PF1 becomes equal to or less than the predetermined threshold may be determined as the principal point of camera 11.

[0268] For example, instead of the detection unit 155, the parameter setting unit 121 may set the principal point of the camera 11 based on the detection result of the stabilization center.

[0269] <Application examples of this technology> This technology can be applied to various cameras equipped with motion sensors and image stabilization functions (e.g., digital still cameras, digital video cameras, surveillance cameras, portable cameras, etc.), as well as devices and systems that adjust the principal points of such cameras. This technology can also be applied to components (e.g., LSIs) and software that process image stabilization functions using motion sensors.

[0270] <<3.Others>> <Example of computer configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0271] FIG. 40 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0272] In the computer 1000, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0273] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0274] The input unit 1006 includes input switches, buttons, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0275] In the computer 1000 configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program recorded in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0276] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0277] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0278] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0279] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0280] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0281] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0282] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0283] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0284] <Example of using the system to set the principal point of the internal parameters> FIG. 41 is a diagram showing an example of using the system for setting the principal points of the internal parameters described above.

[0285] The above-described system can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows:

[0286] ·Digital cameras, mobile devices with camera functions, and other devices that take images for viewing purposes - Devices used for traffic purposes, such as in-vehicle sensors that take pictures of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognizing the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliances according to those gestures. -Medical and healthcare equipment, such as endoscopes and devices that take blood vessel images using infrared light - Security devices such as surveillance cameras for crime prevention and cameras for person authentication Cosmetic devices such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp Sports equipment such as action cameras and wearable cameras for sports purposes Agricultural equipment such as cameras for monitoring the condition of fields and crops

[0287] <Configuration combination example> The present technology can also be configured as follows.

[0288] (1) a detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll angles while the camera is performing the stabilization process; An information processing device. (2) The detection unit detects, as the stabilization center, the center of a trajectory of the feature points in the captured image captured while the camera is rotated in a roll direction while the camera is executing the stabilization process. The information processing device according to (1) above. (3) The detection unit detects the center of the trajectory of the feature points for each rotation while the camera rotates multiple times in the roll direction as the stabilization center, and executes the detection process a predetermined number of times to update a principal point of the internal parameters of the camera to be used in the stabilization process in the next detection process based on the detection result of the stabilization center, and determines the principal point used in the stabilization process in the detection process that minimizes the variation in the positions of the feature points as the principal point of the camera. The information processing device according to (2) above. (4) The detection unit detects the center of the trajectory of the feature points for each rotation while the camera rotates multiple times in the roll direction as the stabilization center, and updates the principal point of the internal parameters of the camera to be used for the stabilization process in the next detection process based on the detection result of the stabilization center. This detection process is repeated until variation in the positions of the feature points becomes equal to or less than a predetermined threshold, and determines the principal point used for the stabilization process in the detection process in which the variation in the positions of the feature points becomes equal to or less than the threshold as the principal point of the camera. The information processing device according to (2) above. (5) The detection unit detects, as the stabilization center, the center of a trajectory of the feature points in the captured image captured while being rotated in a roll direction by a predetermined rotation mechanism. The information processing device according to any one of (2) to (4). (6) The camera is rotated in the roll direction with the center of the lens of the camera and the rotation axis of the rotation mechanism overlapping. The information processing device according to (5) above. (7) The camera is rotated in a roll direction with the optical axis of the camera lens tilted at a predetermined angle with respect to the rotation axis of the rotation mechanism. The information processing device according to (5) or (6). (8) The detection unit detects, as the stabilization center, the center of a trajectory of the feature points in the captured image captured while being rotated in a roll direction by a user's hand. The information processing device according to any one of (2) to (7). (9) The detection unit detects the stabilization center based on positions of the feature points in three or more captured images obtained by capturing images with the camera installed in three or more postures with different roll direction angles. The information processing device according to any one of (1) to (8). (10) The detection unit detects a principal point of the internal parameters of the camera based on the detection result of the stabilization center. The information processing device according to any one of (1) to (9). (11) a parameter setting unit that sets the detected principal point of the camera as a parameter used in the stabilization processing; The information processing device according to (10) further comprises: (12) a stabilization processing unit that executes the stabilization processing based on the parameters set by the parameter setting unit; The information processing device according to (11) further comprises: (13) a user interface processing unit that displays information indicating the center of the photographed image and the positions of the feature points superimposed on the photographed image; The information processing device according to any one of (1) to (12) above further comprises: (14) The detection unit corrects the detection result of the stabilization center based on the amount of deviation between the rotation axis in the roll direction of the camera and the center of the camera coordinate system of the camera. The information processing device according to any one of (1) to (13). (15) The detection unit detects feature points of a component that shows a predetermined pattern in the captured image. The information processing device according to any one of (1) to (14). (16) an offset adjustment unit that adjusts an offset value of the motion sensor based on a measurement value measured by the motion sensor used in the stabilization process while the camera is stationary; The information processing device according to any one of (1) to (15) further comprises: (17) The stabilization center, which is the center of the stabilization process, is detected based on the positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll angles while the camera is performing the stabilization process. Information processing methods. (18) Lenses and An image sensor; A motion sensor; a stabilization processing unit that performs stabilization processing on the captured image obtained by the image sensor based on the measurement values ​​of the motion sensor; a detection unit that detects a stabilization center, which is a center of the stabilization processing, based on positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll direction angles while the stabilization processing is being performed, and detects a principal point of an internal parameter based on the detection result of the stabilization center; a parameter setting unit that sets the detected principal point as a parameter to be used in the stabilization processing; A camera comprising:

[0289] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. [Explanation of symbols]

[0290] 1 Measurement system, 11 Camera, 12 External terminal, 13 Rotation mechanism, 14 Measurement object, 103 Lens, 104 Image sensor, 105 Motion sensor, 106 DSP, 107 Stabilization chip, 121 Parameter setting unit, 122 Mesh setting unit, 123 Offset setting unit, 124 Stabilization processing unit, 125 UI processing unit, 126 Image cropping unit, 152 UI processing unit, 153 Mesh creation unit, 154 Offset adjustment unit, 155 Detection unit, 156 Display operation unit

Claims

1. a detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll angles while the camera is performing the stabilization process; An information processing device.

2. The detection unit detects, as the stabilization center, the center of a trajectory of the feature points in the captured image captured while the camera is rotated in a roll direction while the camera is executing the stabilization process. The information processing device according to claim 1 .

3. The detection unit detects the center of the trajectory of the feature points for each rotation while the camera rotates a plurality of times in the roll direction as the stabilization center, and executes the detection process a predetermined number of times to update a principal point of an internal parameter of the camera to be used in the stabilization process in the next detection process based on the detection result of the stabilization center, and determines the principal point used in the stabilization process in the detection process that minimizes the variation in the positions of the feature points as the principal point of the camera. The information processing device according to claim 2 .

4. The detection unit detects the center of the trajectory of the feature points for each rotation while the camera rotates multiple times in the roll direction as the stabilization center, and repeats the detection process of updating a principal point of the internal parameters of the camera to be used for the stabilization process in the next detection process based on the detection result of the stabilization center until variation in the positions of the feature points becomes equal to or less than a predetermined threshold, and determines the principal point used for the stabilization process in the detection process in which the variation in the positions of the feature points becomes equal to or less than the threshold as the principal point of the camera. The information processing device according to claim 2 .

5. The detection unit detects, as the stabilization center, the center of a trajectory of the feature points in the captured image captured while being rotated in a roll direction by a predetermined rotation mechanism. The information processing device according to claim 2 .

6. The camera is rotated in the roll direction with the center of the lens of the camera and the rotation axis of the rotation mechanism overlapping. The information processing device according to claim 5 .

7. The camera is rotated in a roll direction with the optical axis of the camera lens tilted at a predetermined angle with respect to the rotation axis of the rotation mechanism. The information processing device according to claim 5 .

8. The detection unit detects, as the stabilization center, the center of a trajectory of the feature points in the captured image captured while the camera is rotated in a roll direction by a user's hand. The information processing device according to claim 2 .

9. The detection unit detects the stabilization center based on positions of the feature points in three or more captured images obtained by capturing images with the camera installed in three or more postures with different roll direction angles. The information processing device according to claim 1 .

10. The detection unit detects a principal point of the internal parameters of the camera based on the detection result of the stabilization center. The information processing device according to claim 1 .

11. a parameter setting unit that sets the detected principal point as a parameter used in the stabilization processing; The information processing device according to claim 10 further comprising:

12. a stabilization processing unit that executes the stabilization processing based on the parameters set by the parameter setting unit; The information processing device according to claim 11 further comprising:

13. a user interface processing unit that displays information indicating the center of the photographed image and the positions of the feature points superimposed on the photographed image; The information processing device according to claim 1 further comprising:

14. The detection unit corrects the detection result of the stabilization center based on the amount of deviation between the rotation axis in the roll direction of the camera and the center of the camera coordinate system of the camera. The information processing device according to claim 1 .

15. The detection unit detects feature points of a component that shows a predetermined pattern in the captured image. The information processing device according to claim 1 .

16. an offset adjustment unit that adjusts an offset value of the motion sensor based on a measurement value measured by the motion sensor used in the stabilization process while the camera is stationary; The information processing device according to claim 1 further comprising:

17. The stabilization center, which is the center of the stabilization process, is detected based on the positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll angles while the camera is performing the stabilization process. Information processing methods.

18. Lenses and An image sensor; A motion sensor; a stabilization processing unit that performs stabilization processing on the captured image obtained by the image sensor based on the measurement values ​​of the motion sensor; a detection unit that detects a stabilization center, which is a center of the stabilization process, based on positions of predetermined feature points in a plurality of captured images obtained by capturing images in a plurality of postures with different roll direction angles while the stabilization process is being performed, and detects a principal point of an internal parameter based on the detection result of the stabilization center; a parameter setting unit that sets the detected principal point as a parameter to be used in the stabilization processing; A camera comprising:

Images