Information processing device, information processing method, and measurement system

Abstract

To improve an inspection accuracy as a center of a stabilization.SOLUTION: An inspection part executes inspection processing of a stabilized center as a center of stabilized processing on the basis of a measurement distance of a position of a feature point in a plurality of photographing images obtained by photographing a measurement object in a plurality of postures of which an angle of a roll direction is different in a state where a camera executes stabilized processing, a camera measured by a measurement sensor, and a measurement object. For example, the present technique can be applied to a system for adjusting a main point of an internal parameter of the camera.SELECTED DRAWING: Figure 22

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a measurement system, and more particularly to an information processing device, an information processing method, and a measurement system that can further improve the detection accuracy of a stabilization center. [Background technology]

[0002] Conventionally, a method has been proposed for performing image stabilization using the internal parameters of a camera equipped with an image sensor, a lens, and a gyro sensor (see, for example, Patent Document 1). In the method of Patent Document 1, the smaller the difference between the principal point (Cx, Cy), which is one of the internal parameters of the camera, and the center of the stabilization process (hereinafter referred to as the stabilization center), the higher the accuracy of image stabilization. The stabilization center is the intersection point between the central axis (rotation axis) of the stabilization process and the image coordinate system of the camera.

[0003] Here, stabilization processing refers to processing that suppresses the influence of camera shake on an image by using electronic image stabilization (EIS), or a combination of electronic image stabilization and optical image stabilization (OIS), or mechanical image stabilization (MIS). For example, this method is a method of converting the image shape of a captured image in accordance with the rotation of the camera.

[0004] Methods for transforming image shapes include, for example, congruent transformation, similarity transformation, affine transformation, and projective transformation. Congruent transformation is a transformation method that only allows the rotation of an image, and by adjusting the rotation angle, the images before and after the transformation will match perfectly. Similarity transformation is a transformation method that allows for enlargement and reduction in addition to congruent transformation. Affine transformation is a transformation method that allows for diamond deformation in addition to congruent and similarity transformations. And projective transformation is a transformation method that adds all the functions of congruent transformation, similarity transformation, and affine transformation, and further allows for transformation into any quadrangle.

[0005] In contrast, the principal point (Cx, Cy) of the internal parameters of the camera is set, for example, to the intersection of the optical axis of the lens and the image coordinate system of the camera. More specifically, for example, it is set to the center of the captured image captured by the camera.

[0006] However, in the camera assembly process, there are errors in lens installation, image sensor mounting, and gyro chip mounting, so the central axis of the stabilization process varies for each individual camera. Therefore, the central axis of the stabilization process does not necessarily coincide with the optical axis of the lens.

[0007] In addition, for example, when a camera is attached to a camera while playing sports or when the camera is incorporated into a factory assembly robot, the camera may be subjected to strong shocks or vibrations, causing distortion in the lens or body, which may shift the central axis of the stabilization process. In this case, it becomes necessary to remeasure the stabilization center and correct the principal point (Cx, Cy) of the camera's internal parameters.

[0008] In response to this, the applicant has proposed in Patent Document 2 a method of deriving a central value for stabilization processing by rotating a camera and photographing the central value of a measurement object. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] International Publication No. 2017 / 014071 [Patent Document 2] International Publication No. 2020 / 026527 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in calibration using the method of Patent Document 2, the detection accuracy of the stabilization center decreases, especially when the user rotates the camera by hand, because the distance between the camera and the object to be measured varies.

[0011] The present technology has been made in consideration of such circumstances, and is intended to make it possible to further improve the detection accuracy of the stabilization center. [Means for solving the problem]

[0012] The information processing device of the present technology is an information processing device that has a detection unit that performs detection processing of a stabilization center, which is the center of the stabilization processing, based on the positions of feature points in multiple captured images obtained by photographing a measurement object in multiple postures with different roll direction angles while the camera is performing a stabilization processing, and the measured distance between the camera and the measurement object measured by a distance measuring sensor.

[0013] The information processing method of the present technology is an information processing method that performs a detection process of a stabilization center, which is the center of the stabilization process, based on the positions of feature points in multiple captured images obtained by photographing a measurement object in multiple postures with different roll direction angles while the camera is performing a stabilization process, and the measured distance between the camera and the measurement object measured by a distance measuring sensor.

[0014] The measurement system of the present technology is a measurement system that includes a camera, a ranging sensor, and a detection unit that performs a detection process of a stabilization center, which is the center of the stabilization process, based on the positions of feature points in multiple captured images obtained by photographing a measurement object in multiple postures with different roll direction angles while the camera is performing a stabilization process, and the measured distance between the camera and the measurement object measured by the ranging sensor.

[0015] In this technology, a detection process of the stabilization center, which is the center of the stabilization process, is performed based on the positions of feature points in multiple captured images obtained by photographing the object to be measured in multiple postures with different roll direction angles while the camera is performing a stabilization process, and the measured distance between the camera and the object to be measured measured by a distance measuring sensor. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating calibration using a conventional technique. [Diagram 2] FIG. 1 is a diagram illustrating calibration using the present technology. [Diagram 3] 1 is a block diagram showing an embodiment of a measurement system to which the present technology is applied; [Figure 4] FIG. 2 is a schematic diagram illustrating an example of the hardware configuration of a camera. [Diagram 5] FIG. 1 is a block diagram showing a first embodiment of a camera and an external terminal. [Figure 6] FIG. 11 is a block diagram showing a second embodiment of the camera and an external terminal. [Figure 7] FIG. 11 is a block diagram showing a third embodiment of a camera and an external terminal. [Figure 8] FIG. 13 is a block diagram showing a fourth embodiment of the camera. [Figure 9] FIG. 2 is a block diagram showing a first embodiment of a rotation mechanism. [Figure 10] FIG. 1 is a diagram showing the appearance of a rotation mechanism according to a first embodiment. [Figure 11] FIG. 11 is a block diagram showing a second embodiment of a rotation mechanism. [Figure 12] FIG. 13 is a diagram showing the appearance of a rotation mechanism according to a second embodiment. [Figure 13] FIG. 11 is a plan view showing a rotation mechanism according to a third embodiment. [Figure 14] FIG. 2 is a diagram showing an example of the size of each part of a camera. [Figure 15] FIG. 2 is a diagram showing the positional relationship between a camera coordinate system, an image coordinate system, and a central axis. [Figure 16] 11A and 11B are diagrams illustrating calculation of the amount of deviation between a camera coordinate system and an image coordinate system. [Figure 17] FIG. 2 is a diagram showing an example of a measurement target; [Figure 18] FIG. 2 is a diagram showing an example of a measurement target; [Figure 19] FIG. 2 is a diagram showing an example of a measurement target; [Figure 20] FIG. 2 is a diagram showing an example of a measurement target; [Figure 21] FIG. 2 is a diagram showing an example of a measurement target; [Figure 22] 11 is a flowchart illustrating a calibration process. [Figure 23] FIG. 13 is a diagram showing a display example of a captured image. [Figure 24] FIG. 13 is a diagram showing a display example of a cross mark and characteristic points. [Diagram 25] FIG. 13 is a diagram illustrating an example of a trajectory of feature points. [Figure 26] FIG. 13 is a diagram illustrating an example of a trajectory of feature points. [Figure 27] FIG. 2 is a block diagram showing an example of the configuration of a computer. [Figure 28] FIG. 1 illustrates an example of the use of the system for setting the principal point of an internal parameter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, modes for carrying out the present technology (hereinafter, referred to as embodiments) will be described in the following order.

[0018] 1. Issues with conventional technologies and overview of this technology 2. Configuration of measurement system using this technology 3. Examples of measurement objects 4. Calibration process 5.Other

[0019] <1. Issues with conventional technologies and overview of this technology> Action cameras are designed to be worn on the user's body to capture outdoor and sports scenes, but it is not easy to replace the lenses.

[0020] In addition, in a camera with interchangeable lenses, the calibration values ​​change when the lens is removed, so conventional parameters cannot be used, and calibration must be performed every time the lens is changed.

[0021] 1, calibration using the method of Patent Document 2 is premised on the fact that the distance L between the camera CAM and the measurement object OM is accurate. However, when calibration is performed while rotating the camera around the rotation axis AR by the user's hand, the camera CAM moves back and forth due to the movement of the user's hand, and the distance L between the camera CAM and the measurement object OM varies, which reduces the detection accuracy of the stabilization center.

[0022] In response to this, in this technology, as shown in Fig. 2, a distance measuring sensor RS is newly provided and linked with the camera CAM to measure the distance L between the camera CAM and the measurement object OM each time an image is captured, thereby realizing the derivation of the center value of the stabilization process. This allows the distance L to be measured accurately even if the camera CAM moves back and forth due to the movement of the user's hand, making it possible to further improve the detection accuracy of the stabilization center.

[0023] The configuration of a measurement system to which the present technology is applied will be described below.

[0024] 2. Configuration of the measurement system using this technology (Example of measurement system configuration) FIG. 3 is a block diagram showing an embodiment of a measurement system to which the present technology is applied.

[0025] 3 includes a camera 11, an external terminal 12, a rotation mechanism 13, and a measurement object 14. The measurement system 1 is a system that detects the stabilization center of the camera 11, and sets the principal point (Cx, Cy) of the internal parameters of the camera 11 based on the detection result of the stabilization center.

[0026] In the following description, unless otherwise specified, the principal point of the camera 11 does not represent the principal point of the optical center of the camera 11 but represents the principal point (Cx, Cy) of the internal parameters of the camera 11.

[0027] The camera 11 is attached to a rotation mechanism 13 and captures an image of a measurement object 14 .

[0028] The external terminal 12 is configured by, for example, a general-purpose terminal such as a smartphone, a tablet, a PC (Personal Computer), a dedicated terminal for camera calibration, etc. The external terminal 12 performs wireless or wired communication with the camera 11.

[0029] The rotation mechanism 13 is a mechanism for rotating the camera 11 in a roll direction around a rotation axis AR. Here, the roll direction of the camera 11 is the direction of rotation around the optical axis of the lens 103 (the Z axis of the camera coordinate system). Note that the rotation mechanism 13 does not necessarily have to be provided in the measurement system 1. In this case, the user rotates the camera 11 in the roll direction with his or her own hand.

[0030] The measurement object 14 is an object that provides feature points used to detect the stabilization center of the camera 11. The camera 11 and the measurement object 14 are placed opposite to the camera 11 such that the feature points of the measurement object 14 do not go outside the viewing angle Φ of the camera 11 when the camera 11 rotates around the rotation axis AR.

[0031] The measurement system 1 further comprises a distance measurement sensor 15 .

[0032] The distance measuring sensor 15 is configured by a LiDAR (Light Detection and Ranging) using laser light, a RADAR (Radio Detecting and Ranging) using electromagnetic waves corresponding to millimeter waves, an ultrasonic sensor using ultrasonic waves, a stereo camera, etc. The distance measuring sensor 15 is provided in the close vicinity of the camera 11 and measures the distance between the camera 11 and the measurement object 14.

[0033] Hereinafter, the distance between the camera 11 and the measurement object 14 (measurement distance) is defined as L.

[0034] (Camera configuration example) FIG. 4 is a diagram illustrating an example of the hardware configuration of the camera 11. As shown in FIG.

[0035] The camera 11 includes a housing 101, a support substrate 102, a lens 103, an image sensor 104, a motion sensor 105, a DSP (Digital Signal Processor) 106, and a stabilization chip 107.

[0036] The support substrate 102 is placed in the housing 101 so as to be approximately perpendicular to the optical axis AC of the lens 103 .

[0037] The lens 103 is disposed at a predetermined position in the housing 101 and forms an image of a subject on the light receiving surface of the image sensor 104 .

[0038] The image sensor 104 is disposed on the support substrate 102 so as to be approximately perpendicular to the optical axis AC of the lens 103. The image sensor 104 captures an image of a subject formed by the lens 103, and generates a captured image.

[0039] The motion sensor 105 is disposed on the surface of the support substrate 102 opposite to the surface on which the image sensor 104 is disposed. The motion sensor 105 includes, for example, a gyro sensor and an acceleration sensor, and detects the angular velocity and acceleration of the camera 11.

[0040] The DSP 106 is installed on the inner surface of the housing 101. The DSP 106 executes various types of image processing on the captured image.

[0041] The stabilization chip 107 is installed on the inner surface of the housing 101. The stabilization chip 107 executes stabilization processing on the captured image.

[0042] (Example of camera and external device configuration) FIG. 5 is a diagram showing an example of the functional configuration of the camera 11 and the external terminal 12 shown in FIG.

[0043] In addition to the above-mentioned lens 103, image sensor 104, and motion sensor 105, camera 11 includes a parameter setting unit 121, a mesh setting unit 122, an offset setting unit 123, a stabilization processing unit 124, a UI (User Interface) processing unit 125, a signal processing unit 126, an image cropping unit 127, and an I / O (Input / Output) interface unit 128. The image sensor 104, the parameter setting unit 121, the mesh setting unit 122, the offset setting unit 123, the stabilization processing unit 124, the UI processing unit 125, the signal processing unit 126, the image cropping unit 127, and the I / O interface unit 128 are connected to each other via a bus.

[0044] The image sensor 104 supplies captured image data corresponding to the captured image to a stabilization processor 124 and a signal processor 126 .

[0045] The parameter setting unit 121 receives input information including the internal parameters of the camera 11 input at the external terminal 12, and principal point information indicating the detection result of the principal point of the camera 11 by the detection unit 155 of the external terminal 12, from the external terminal 12 via the I / O interface unit 128. The parameter setting unit 121 sets the internal parameters of the camera 11 based on the input information and the principal point information. The parameter setting unit 121 supplies parameter information related to the set internal parameters to the stabilization processing unit 124. In addition, the parameter setting unit 121 transmits the parameter information to the external terminal 12 via the I / O interface unit 128.

[0046] The mesh setting unit 122 receives mesh information on the mesh of the camera coordinate system generated by the mesh creation unit 153 of the external terminal 12 from the external terminal 12 via the I / O interface unit 128. The mesh setting unit 122 sets the mesh of the camera coordinate system by storing the mesh information. In addition, the mesh setting unit 122 supplies the mesh information to the image cropping unit 127.

[0047] The offset setting unit 123 receives offset information indicating the offset value of the motion sensor 105 adjusted by the offset adjustment unit 154 of the external terminal 12 from the external terminal 12 via the I / O interface unit 128. The offset setting unit 123 sets the offset value of the motion sensor 105 based on the offset information.

[0048] The offset setting unit 123 generates motion sensor information indicating the angular velocity and acceleration of the camera 11 measured by the motion sensor 105, and supplies the motion sensor information to the stabilization processing unit 124. In addition, the offset setting unit 123 transmits the motion sensor information to the external terminal 12 via the I / O interface unit 128.

[0049] The stabilization processing unit 124 is realized by, for example, the stabilization chip 107 in Fig. 4. The stabilization processing unit 124 executes stabilization processing on the captured image based on the parameter information and the motion sensor information. As described above, the stabilization processing is processing (homography processing) for projectively transforming the captured image in accordance with the rotation of the camera 11. The stabilization processing unit 124 supplies the captured image data corresponding to the captured image after the stabilization processing to the signal processing unit 126.

[0050] A UI (User Interface) processing unit 125 executes processing related to the user interface of the camera 11 .

[0051] The signal processing unit 126 is realized by, for example, the DSP 106 in Fig. 4. The signal processing unit 126 executes various signal processes on the captured image before or after the stabilization process. The signal processing unit 126 transmits captured image data corresponding to the captured image before the stabilization process to the external terminal 12 via the I / O interface unit 128. The signal processing unit 126 supplies captured image data corresponding to the captured image after the stabilization process to the image cropping unit 127.

[0052] The image cropping unit 127 crops out an image of a predetermined size from the captured image after stabilization processing based on the mesh information. The image cropping unit 127 transmits cropped image data corresponding to the cropped captured image (hereinafter referred to as cropped image) to the external terminal 12 via the I / O interface unit 128.

[0053] The I / O interface unit 128 is connected to the external terminal 12 by wire or wirelessly, and communicates with the external terminal 12. When the I / O interface unit 128 is connected to the external terminal 12 by wire, it includes, for example, a commonly used external connection connector.

[0054] The camera 11 further includes an I / O interface unit 129. The I / O interface unit 129 is also connected to other functional blocks constituting the camera 11 via a bus.

[0055] The I / O interface unit 129 is connected to a sensor housing 180 (described later) by wire or wirelessly, and communicates with the sensor housing 180. When the I / O interface unit 129 is connected to the sensor housing 180 by wire, it includes, for example, a commonly used external connection connector.

[0056] The external terminal 12 includes an I / O interface unit 151, a UI processing unit 152, a mesh creation unit 153, an offset adjustment unit 154, a detection unit 155, and a display operation unit 156. The I / O interface unit 151, the UI processing unit 152, the mesh creation unit 153, the offset adjustment unit 154, the detection unit 155, and the display operation unit 156 are connected to each other via a bus.

[0057] The I / O interface unit 151 is connected to the camera 11 by wire or wirelessly, and communicates with the camera 11. When the I / O interface unit 151 is connected to the camera 11 by wire, it includes, for example, a commonly used external connection connector.

[0058] The UI processing unit 152 executes processing related to the user interface of the external terminal 12. For example, the UI processing unit 152 controls the display of the display operation unit 156.

[0059] The mesh creation unit 153 receives parameter information from the camera 11 via the I / O interface unit 151. Based on the parameter information, the mesh creation unit 153 creates a mesh in the camera coordinate system so as to have a desired format for the camera 11. The mesh creation unit 153 generates mesh information on the created mesh and transmits it to the camera 11 via the I / O interface unit 151.

[0060] The offset adjustment unit 154 receives motion sensor information from the camera 11 via the I / O interface unit 151. The offset adjustment unit 154 adjusts the offset value of the motion sensor 105 based on the motion sensor information. The offset adjustment unit 154 generates offset information indicating the offset value of the motion sensor 105 after the adjustment, and transmits it to the camera 11 via the I / O interface unit 151.

[0061] The detection unit 155 receives captured image data and cropped image data from the camera 11 via the I / O interface unit 151. The detection unit 155 detects feature points in the captured image data and cropped image data, and supplies information on the detected feature points to the UI processing unit 152. The detection unit 155 detects a stabilization center based on the captured image data and cropped image data, and detects the principal point of the camera 11 based on the detection result of the stabilization center. The detection unit 155 generates principal point information indicating the detection result of the principal point of the camera 11, and transmits it to the camera 11 via the I / O interface unit 151.

[0062] The display operation unit 156 is configured by, for example, a combination of a display device and an operation member, such as a touch panel or a monitor and a keyboard. The display operation unit 156 is used, for example, to display various information and to input operation signals and various information to the camera 11 and the external terminal 12.

[0063] For example, under the control of the UI processing unit 152, the display operation unit 156 displays the captured image, the cropped image, the feature points and optical flow of the captured image and the cropped image, the measurement results of the motion sensor 105, and the like.

[0064] For example, the display operation unit 156 displays a setting screen under the control of the UI processing unit 152, and enables various pieces of information to be set, created, checked, etc. Information that can be set, created, checked, etc. on the setting screen includes, for example, internal parameters of the camera 11, mesh information of the camera coordinate system, setting information related to feature point detection, setting information related to stabilization processing, setting information related to stabilization center detection processing, setting information related to signal processing, and setting information related to shooting (for example, aperture, shutter speed, gain, AE (automatic exposure), etc.). In addition, information that can be set, created, checked, etc. on the setting screen includes, for example, information on each part of the camera 11 and the external terminal 12.

[0065] The display operation unit 156 supplies input information relating to an input operation signal and various information to each unit of the external terminal 12. In addition, the display operation unit 156 transmits the input information to the camera 11 via the I / O interface unit 151.

[0066] A sensor housing 180 is attached to the outside of the camera 11 .

[0067] The sensor housing 180 includes a distance measurement processing unit 181 and an I / O interface unit 182 in addition to the distance measurement sensor 15 described above.

[0068] The distance measurement sensor 15 measures the distance between the camera 11 and the measurement object 14, and supplies sensor data corresponding to the measured distance L to the distance measurement processing unit 181.

[0069] The distance measurement processing unit 181 performs various data processing on the sensor data from the distance measurement sensor 15 to generate distance information indicating the measured distance L. The distance measurement processing unit 181 outputs the distance information to the camera 11 via the I / O interface unit 182.

[0070] The I / O interface unit 182 is connected to the camera 11 by wire or wirelessly, and communicates with the camera 11. When the I / O interface unit 182 is connected to the camera 11 by wire, it includes, for example, a commonly used external connection connector.

[0071] The distance information output from the sensor housing 180 to the camera 11 is transmitted to the external terminal 12 via the I / O interface unit 128. The detection unit 155 of the external terminal 12 receives the distance information indicating the measured distance L measured by the distance measuring sensor 15. The detection unit 155 detects the stabilization center based on the captured image data and the cropped image data as well as the measured distance L indicated by the distance information, and detects the principal point of the camera 11 based on the detection result of the stabilization center.

[0072] The distance measurement sensor 15 and the distance measurement processing unit 181 may take a form other than being externally attached to the camera 11 as the sensor housing 180 .

[0073] For example, as shown in FIG. 6, the distance measurement sensor 15 and the distance measurement processing unit 181 may be built into the camera 11.

[0074] 7, the camera 11 may be provided with an image sensor 104RS incorporating a distance measurement sensor instead of the image sensor 104. In this case, the distance measurement sensor may be an indirect time of flight (iToF) sensor or a direct time of flight (dToF) sensor. The distance measurement processing unit 181 may be connected to the image sensor 104RS via a bus, or may be built into the image sensor 104RS.

[0075] Furthermore, the configurations of the camera 11 and the external terminal 12 can be changed as appropriate. For example, the division of functions between the camera 11 and the external terminal 12 can be changed.

[0076] For example, some of the components of the external terminal 12 can be provided in the camera 11 instead of the external terminal 12. Also, for example, as shown in Fig. 8, all of the components of the external terminal 12 can be provided in the camera 11. In this case, it becomes possible to delete the external terminal 12 from the measurement system 1.

[0077] In the above-mentioned configuration example, for example, it is possible to integrate the mesh setting unit 122 and the mesh creating unit 153 into one. For example, it is possible to integrate the offset setting unit 123 and the offset adjusting unit 154 into one. Also, in the above-mentioned configuration example, an operation unit and a display unit may be provided in the camera 11.

[0078] In the following, an example in which the camera 11 and the external terminal 12 are configured separately will be described.

[0079] (Example of rotation mechanism configuration) Next, a configuration example of the rotation mechanism 13 in FIG. 3 will be described with reference to FIGS.

[0080] FIG. 9 is a block diagram showing the rotation mechanism 13 according to the first embodiment, and FIG. 10 is a diagram showing the external appearance of the rotation mechanism 13 according to the first embodiment.

[0081] The rotation mechanism 13 includes a rotation jig 201, a rotation shaft 202, and an attachment jig 203. The rotation jig 201 and the attachment jig 203 are connected via the rotation shaft 202.

[0082] The rotating jig 201 is a jig for rotating the rotating shaft 202. For example, the rotating jig 201 is configured by a handle or the like, which will be described later.

[0083] The mounting jig 203 is a jig for mounting the camera 11 to the rotation mechanism 13 .

[0084] For example, when a user rotates a rotation shaft 202 using a rotation jig 201, an attachment jig 203 connected to the rotation shaft 202, and the camera 11 and distance measurement sensor 15 attached to the attachment jig 203 also rotate.

[0085] That is, as shown in Figure 10, the camera 11 and the distance measurement sensor 15 are rotated in the roll direction around the rotation axis AR by the rotation mechanism 13, while the distance measurement sensor 15 measures the measurement distance L between the camera 11 and the object to be measured 14.

[0086] Fig. 11 is a block diagram showing a second embodiment of the rotation mechanism 13, and Fig. 12 is a diagram showing the appearance of the second embodiment of the rotation mechanism 13. In the figures, parts corresponding to those of the rotation mechanism 13 in Figs. 9 and 10 are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0087] The rotation mechanism 13 in FIG. 11 has a configuration in which an angle jig 204 is added to the rotation mechanism 13 in FIG.

[0088] Angle jig 204 is provided between rotation jig 201 and mounting jig 203. Angle jig 204 is a jig that can change the angles in the pitch direction and the yaw direction of mounting jig 203. By adjusting the angles in the pitch direction and the yaw direction of angle jig 204, the angles in the pitch direction and the yaw direction of camera 11 and distance measuring sensor 15 attached to mounting jig 203 can be adjusted.

[0089] 12, the camera 11 and the distance measuring sensor 15 are rotated in the roll direction about the rotation axis AR by the angle jig 204 while being tilted at a predetermined angle with respect to the rotation axis AR of the rotation mechanism 13, and the distance measuring sensor 15 measures the measurement distance L between the camera 11 and the measurement object 14. In this case, the camera 11 and the distance measuring sensor 15 can be tilted within the directivity range of the distance measuring sensor 15.

[0090] This makes it possible to increase the amount of change in the motion sensor 105 (angular velocity and acceleration of the camera 11) when the camera 11 is rotated by the rotation mechanism 13, thereby improving the detection accuracy of the stabilization center.

[0091] Fig. 13 is a plan view showing a third embodiment of the rotation mechanism 13. In the drawing, parts corresponding to those in the rotation mechanism 13 in Fig. 9 and Fig. 10 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0092] The rotation mechanism 13 shown in FIG. 13 includes a rotation shaft 202, a mounting jig 203, a mounting stand 221, a handle 222, a bearing 223-1, and a bearing 223-2.

[0093] The rotating shaft 202 is fixed and rotatable by bearings 223-1 and 223-2 installed on the installation base 221. A handle 222 is attached to one end of the rotating shaft 202, and a mounting jig 203 is attached to the other end of the rotating shaft 202. The camera 11 and the distance measuring sensor 15 are attached to the mounting jig 203.

[0094] By rotating the handle 222, the user can rotate the camera 11 and the distance measuring sensor 15 in the roll direction around the rotation axis 202.

[0095] Although not shown, an angle jig 204 can be installed between the rotation shaft 202 and the mounting jig 203 .

[0096] (Relationship between the misalignment of the lens center and rotation axis and stabilization processing) When the center of lens 103 of camera 11 and rotation axis AR of rotation mechanism 13 are misaligned by a distance t, a misalignment of distance t also occurs between the rotation axis AR and the optical axis of lens 103.

[0097] In this case, the stabilization center cannot be accurately detected in the stabilization center detection process described below.

[0098] Therefore, it is desirable to adjust at least one of the positions of the camera 11 and the mounting jig 203 so that the center of the lens 103 coincides with the rotation axis AR. Alternatively, it is desirable to use a mounting jig 203 such that the center of the lens 103 coincides with the rotation axis AR when the camera 11 is attached. For example, the amount of deviation between the center of the camera coordinate system and the center of the image coordinate system and the rotation axis AR may be corrected.

[0099] Here, an example of a method for calculating the amount of deviation between the center of the camera coordinate system and the center of the image coordinate system, and the rotation axis AR, will be described with reference to FIGS.

[0100] Hereinafter, as shown in Fig. 14, the center of rotation when the camera 11 rotates by rotating the rotation mechanism 13 is defined as CR. The center of rotation CR overlaps with the rotation axis AR of the rotation mechanism 13. Hereinafter, the height (vertical distance) between the center of rotation CR and the center CL of the lens 103 is defined as H LENS Let the width (horizontal distance) be W LENS In the following, the height of the camera 11 is H CAM Let the width be W CAM Let us assume that.

[0101] Hereinafter, as shown in FIG. 15, the amount of deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR is defined as t. Hereinafter, the amount of deviation in the height (vertical) direction of the amount of deviation t is defined as t h Let the horizontal deviation be t w Hereinafter, the point where the measurement object 14 intersects with the rotation axis AR is referred to as PC. Hereinafter, the amount of deviation between the center of the image coordinate system and the rotation axis AR is referred to as f t In the following, the deviation amount f t Among these, the amount of deviation in the height (vertical) direction is f th Let the horizontal deviation be f tw Let us assume that.

[0102] 16A shows the relationship between the amount of deviation t in the camera coordinate system and the measurement distance L between the camera 11 and the measurement target 14. FIG. 20B shows the relationship between the amount of deviation f t and the focal length f of the lens 103.

[0103] Here, the triangle in FIG. 20A and the triangle in FIG. 20B are similar. In addition, the amount of deviation t h and the amount of deviation t w is expressed by the following equations (1) and (2).

[0104] t h =H LENS (1) t w =W LENS (2)

[0105] Therefore, the displacement in the image coordinate system is f th and the deviation f tw is expressed by the following equations (3) and (4).

[0106] f th =f×f t / L=f×H LENS / L...(3) f tw =f×f t / L=f×W LENS / L (4)

[0107] Here, for example, H LENS =1mm,W LENS = 1 mm, L = 2 m (2000 mm), the focal length of the camera 11 converted into pixels is f = 1000 pixels, the amount of deviation f th and the deviation f tw is 0.5 pixels as shown in the following expressions (5) and (6).

[0108] f th =1000×1 / 2000=0.5 (5) f tw =1000×1 / 2000=0.5 (6)

[0109] For example, the detection unit 155 of the external terminal 12 may detect LENS , W LENS , the measurement distance L, and the focal length f are obtained from the outside, and the displacement amount f of the image coordinate system is calculated. th and the deviation f tw After detecting the stabilization center in a process to be described later, the detection unit 155 calculates the deviation amount f th and the deviation f tw The detection value is corrected by the amount of

[0110] This corrects the error in the stabilization center caused by the amount of deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR of the rotation mechanism 13, improving the detection accuracy of the stabilization center. Furthermore, the amount of deviation f of the image coordinate system is calculated using the measured distance L measured by the distance measuring sensor 15. th and the deviation f tw Since the above calculation is performed, it is possible to further improve the detection accuracy of the stabilization center even when the camera 11 moves back and forth due to the movement of the user's hand, causing the distance between the camera 11 and the measurement object 14 to vary.

[0111] The distance between the camera 11 and the measurement object 14 may be set to an infinity equivalent. th and the deviation f tw becomes essentially 0, making correction of the detection value of the stabilization center unnecessary.

[0112] <3. Examples of measurement objects> An example of the measurement object 14 will be described with reference to FIGS.

[0113] FIG. 17 shows a first example of a measurement object 14 exhibiting a pattern including a single feature point.

[0114] The measurement object 14 shown in Figure 17 has a circular mark in the center when viewed from the front, as shown in Figure A, and when viewed from the side (cross-section), the mark is formed by a cone-shaped protrusion, as shown in Figure B.

[0115] FIG. 18 shows a second example of a measurement object 14 exhibiting a pattern including a single feature point.

[0116] The measurement object 14 shown in Figure 18 has a set of lattice patterns (a pattern in which rectangles of two colors are arranged alternately) when viewed from the front, as shown in Figure A, and when viewed from the side (cross-section), rectangles of one color (black in the figure) are formed by box-shaped protrusions, as shown in Figure B, or when viewed from the side (cross-section), rectangles of the other color (white in the figure) are formed by holes, as shown in Figure C.

[0117] FIG. 19 shows a first example of a measurement object 14 showing a pattern including a plurality of feature points.

[0118] The measurement object 14 shown in Figure 19 has multiple circular marks arranged in a grid pattern when viewed from the front, as shown in Figure A, and when viewed from the side (cross-section), the marks are formed by cone-shaped protrusions, as shown in Figure B.

[0119] FIG. 20 shows a second example of the measurement object 14 showing a pattern including a plurality of feature points.

[0120] The measurement object 14 shown in Figure 20 has multiple lattice patterns when viewed from the front, as shown in Figure A, and is formed by rectangles of one color (black in the figure) formed by box-shaped protrusions when viewed from the side (cross-section), as shown in Figure B, or is formed by rectangles of the other color (white in the figure) formed by holes when viewed from the side (cross-section), as shown in Figure C.

[0121] As described above, the object to be measured 14 in the measurement system to which the present technology is applied needs to measure the distance using a distance measuring sensor 15, so it is not formed only from a plate-shaped member as disclosed in Patent Document 2, but at least a part of it is formed with an uneven shape or holes.

[0122] On the other hand, in a measurement system to which the present technology is applied, an external object can also be used as the measurement target 14, as in Patent Document 2.

[0123] FIG. 21 shows an example in which an external object is used as the measurement target 14.

[0124] 21A shows an example in which an outdoor house or building is used as the measurement object 14, and FIG. 21B shows an example in which an outdoor sign or utility pole is used as the measurement object 14. The external object shown in FIG. 21 also has a part formed with an uneven shape.

[0125] In addition, in addition to the example shown in Figure 21, the external object used for the measurement target 14 can be an object that can detect feature points, has feature points whose positions are stationary, and has a part formed with an uneven shape or a hole.

[0126] <4. Calibration process> The calibration process executed by the measurement system 1 will be described with reference to the flowchart of Fig. 22. Note that detailed descriptions of processes similar to the calibration process disclosed in Patent Document 2 will be omitted as appropriate.

[0127] Before the calibration process, an environment for detecting the stabilization center is constructed as described with reference to Fig. 3. That is, the camera 11 is attached to the rotation mechanism 13, and the measurement target object 14 is placed so as to face the camera 11.

[0128] In step S1, the measurement system 1 captures and displays an image of the measurement target 14.

[0129] Specifically, the image sensor 104 of the camera 11 captures an image of the measurement target 14 via the lens 103, and supplies the obtained captured image data to the signal processing unit 126. The signal processing unit 126 executes various signal processing on the captured image, and transmits the captured image data after the signal processing to the external terminal 12 via the I / O interface unit 128.

[0130] In response to this, the I / O interface unit 151 of the external terminal 12 receives the captured image data from the camera 11. The I / O interface unit 151 supplies the captured image data to each unit of the external terminal 12. The display operation unit 156 displays the captured image in which the measurement object 14 is captured, based on the captured image data. As a result, for example, as shown in Fig. 23, the captured image in which the measurement object 14 is captured is displayed on the display operation unit 156.

[0131] In step S2, the external terminal 12 acquires the internal parameters of the camera 11. For example, the user inputs the internal parameters of the camera 11 via the display operation unit 156 of the external terminal 12.

[0132] The internal parameters of the camera 11 include correction coefficients (distortion parameters) k1 to k4, p1, and p2 in addition to the principal point (Cx, Cy) and focal length (fx, fy) of the camera 11. These internal parameters are used for distortion correction of the lens 103, stabilization processing, and the like.

[0133] For example, the focal lengths (fx, fy) and the correction coefficients k1 to k4, p1, and p2 are specified or guaranteed by the lens manufacturer. Alternatively, parameters measured using a lens parameter measuring device are used.

[0134] For example, the coordinates of the center of the captured image are provisionally set as the principal point (Cx, Cy).

[0135] The display operation unit 156 generates input information including the input internal parameters, and supplies the input information to each unit of the external terminal 12. The display operation unit 156 also transmits the input information to the camera 11 via the I / O interface unit 151.

[0136] In response to this, the parameter setting unit 121 of the camera 11 receives the input information via the I / O interface unit 128. The parameter setting unit 121 sets internal parameters of the camera 11 based on the input information. The parameter setting unit 121 generates parameter information related to the set internal parameters, and supplies the parameter information to the stabilization processing unit 124.

[0137] In step S3, the measurement system 1 creates and sets a mesh of the camera coordinate system.

[0138] Specifically, the mesh creating unit 153 of the external terminal 12 creates a mesh in the camera coordinate system based on the internal parameters of the camera 11 acquired in the process of step S2.

[0139] Any method can be applied to create the mesh of the camera coordinate system. For example, the methods disclosed in Patent Document 1, JP 2013-134088 A, WO 2020 / 039747, etc. can be applied.

[0140] Also, for example, a plurality of types of meshes may be prepared in advance, and an appropriate mesh may be selected from among them based on the internal parameters of the camera 11.

[0141] The mesh creation unit 153 generates mesh information regarding the created mesh, and transmits it to the camera 11 via the I / O interface unit 151 .

[0142] In response to this, the mesh setting unit 122 of the camera 11 receives the mesh information from the external terminal 12 via the I / O interface unit 128. The mesh setting unit 122 sets a mesh in the camera coordinate system based on the mesh information. The mesh setting unit 122 generates mesh information on the set mesh and supplies it to the image cropping unit 127.

[0143] In step S4, the display operation unit 156 of the external terminal 12 displays the center of the captured image under the control of the UI processing unit 152. For example, a cross mark, which is information indicating the position of the center of the captured image, is superimposed on the captured image displayed on the display operation unit 156.

[0144] On the other hand, if the center of the captured image and the center of the measurement object 14 are misaligned, the user adjusts the position of the measurement object 14 as necessary so that the center of the captured image overlaps the center of the measurement object 14.

[0145] In step S5, the external terminal 12 detects and displays feature points of the captured image.

[0146] Specifically, the detection unit 155 of the external terminal 12 detects feature points of the captured image.

[0147] Note that any method can be applied to detect feature points, such as Moravec's corner detection algorithm, Harris, Stephens, and Plessey's corner detection algorithm, multi-scale Harris operator, Shi and Tomasi's method, curvature-based method, Laplacian-Gaussian and derivative methods, Wang and Brady's method, Trajkovic and Hedley's method, FAST feature detection, etc.

[0148] The detection unit 155 supplies information about the detected feature points to the UI processing unit 152. The display operation unit 156, under the control of the UI processing unit 152, superimposes information indicating the positions of the feature points on the captured image being displayed.

[0149] As a result, the center of the measurement object 14 in the captured image is detected as the feature point PF, and, for example, as shown in Figure 24, information indicating the position of the detected feature point PF is superimposed on the captured image on which the cross mark displayed on the display operation unit 156 is superimposed.

[0150] For example, when multiple feature points are detected and displayed, the user may use the display operation unit 156 to select one feature point to be used for detecting the stabilization center and not display the other feature points.

[0151] Furthermore, for example, if a feature point is not detected or the feature point detection result is unstable, the user may use the display operation unit 156 to change the feature point detection method or detection parameters.

[0152] Next, the user uses the rotation mechanism 13 to rotate the camera 11 in the roll direction.

[0153] In response to this, for example, the display operation unit 156 superimposes the trajectory of the feature points PF on the captured image under the control of the UI processing unit 152. Then, the user confirms that the feature points PF do not disappear but continue to be displayed on the display operation unit 156 even if the camera 11 is rotated. In other words, the user confirms that the feature points PF do not go out of the viewing angle Φ of the camera 11 even if the camera 11 is rotated.

[0154] If rotating the camera 11 causes the characteristic point PF to disappear from the display operation unit 156, the user adjusts the position of the measurement object 14 so that the characteristic point PF does not go outside the viewing angle Φ of the camera 11.

[0155] Furthermore, for example, the user adjusts the position of the measurement object 14 as necessary so that the center of the circle indicating the locus of the feature points PF approximately overlaps with the center of the captured image.

[0156] In step S6, the distance measurement sensor 15 measures the distance between the camera 11 and the measurement object 14.

[0157] Next, in step S7, the camera 11 starts the stabilization process.

[0158] Specifically, for example, the user uses an operation unit (not shown) of the camera 11 to perform an operation to set the image stabilization function to ON.

[0159] In response to this, the UI processing unit 125 sets the image stabilization function to ON, and notifies each unit of the camera 11 that the image stabilization function has been set to ON.

[0160] The stabilization processing unit 124 executes the stabilization process. For example, the stabilization processing unit 124 executes the stabilization process by transforming the coordinates of the captured image using a rotation matrix and a projective transformation matrix.

[0161] The stabilization processing unit 124 supplies the captured image data after the stabilization processing to the image cropping unit 127 .

[0162] The image cropping unit 127 crops out an image of a predetermined size from the captured image based on the mesh information supplied from the mesh setting unit 122 .

[0163] The image cropping unit 127 transmits cropped image data corresponding to the cropped image cropped from the captured image to the external terminal 12 via the I / O interface unit 128.

[0164] In response to this, the I / O interface unit 151 of the external terminal 12 receives the cropped image data and supplies it to each unit of the external terminal 12. The detection unit 155 detects feature points of the cropped image based on the cropped image data, and supplies information on the detected feature points to the UI processing unit 152.

[0165] The display operation unit 156 displays the cut-out image based on the cut-out image data under the control of the UI processing unit 152. In addition, the display operation unit 156 superimposes information indicating the positions of a cross mark and a feature point on the cut-out image under the control of the UI processing unit 152.

[0166] As a result, a cut-out image including the measurement object 14 and on which the cross mark and information indicating the positions of the characteristic point PF are superimposed is displayed on the display operation unit 156.

[0167] In step S8, the external terminal 12 starts tracking the feature points.

[0168] Specifically, for example, the user uses the display operation unit 156 to perform an operation to set the optical flow function to ON.

[0169] In response to this, the UI processing unit 152 sets the optical flow function to ON and notifies each unit of the external terminal 12 that the optical flow function has been set to ON. The detection unit 155 starts tracking feature points in the cropped image and supplies the UI processing unit 152 with information on the tracking results of the feature points.

[0170] Next, the user uses the rotation mechanism 13 to rotate the camera 11 in the roll direction.

[0171] In response to this, the display operation unit 156 superimposes the trajectory of the feature points PF on the cut-out image under the control of the UI processing unit 152. Then, the user confirms that the feature points PF do not disappear but continue to be displayed on the display operation unit 156 even if the camera 11 is rotated. In other words, the user confirms that the feature points PF do not go out of the cut-out image even if the camera 11 is rotated.

[0172] If the rotation of the camera 11 causes the feature point PF to extend beyond the cut-out image, the user adjusts the position of the measurement object 14 so that the feature point PF does not extend beyond the cut-out image.

[0173] Furthermore, the user adjusts the position of the measurement object 14 so that the center of the locus of the feature points PF substantially overlaps with the center of the cut-out image.

[0174] In step S9, the offset setting unit 123 of the camera 11 determines whether or not the offset of the motion sensor 105 has been corrected. If it is determined that the offset of the motion sensor 105 has not been corrected, the process proceeds to step S10.

[0175] In step S10, the measurement system 1 corrects the offset of the motion sensor 105.

[0176] For example, the user holds the camera 11 still for a predetermined time (e.g., 10 seconds) or more. The motion sensor 105 measures the angular velocity of the motion sensor 105 during the stationary period, i.e., the zero point of the angular velocity of the motion sensor 105. The angular velocity of the motion sensor 105 is represented by, for example, angular velocity Gx, angular velocity Gy, and angular velocity Gz on the X-axis, Y-axis, and Z-axis of the camera coordinate system.

[0177] The offset setting unit 123 generates motion sensor information including the measurement results of the angular velocity of the motion sensor 105 during the stationary period, and transmits the generated motion sensor information to the external terminal 12 via the I / O interface unit 128 .

[0178] In response to this, the offset adjustment unit 154 of the external terminal 12 receives the motion sensor information from the camera 11 via the I / O interface unit 151.

[0179] The offset adjustment unit 154 calculates the average of each measurement value of the angular velocities Gx, Gy, Gz during the stationary period as the zero point of the angular velocities Gx, Gy, Gz. The offset adjustment unit 154 calculates offset values ​​for correcting the zero point values ​​of the angular velocities Gx, Gy, Gz of the motion sensor 105 to 0. The offset adjustment unit 154 transmits offset information indicating the calculated offset values ​​to the camera 11 via the I / O interface unit 151.

[0180] In response to this, the offset setting unit 123 of the camera 11 receives the offset information from the external terminal 12 via the I / O interface unit 128. The offset setting unit 123 sets an offset value for the angular velocity of the motion sensor 105 based on the offset information.

[0181] Although a detailed description is omitted, the offset values ​​of the acceleration of the motion sensor 105 on the X-axis, Y-axis, and Z-axis of the camera coordinate system are set in the same manner as the offset values ​​of the angular velocity.

[0182] Then, the process proceeds to step S11.

[0183] On the other hand, if it is determined in step S9 that the offset of the motion sensor 105 has been corrected, the process of step S10 is skipped and the process proceeds to step S11.

[0184] In step S11, the measurement system 1 executes a process of detecting the center of stabilization.

[0185] For example, the detection unit 155 sets the position of the feature point PF in the image coordinate system of the cut-out image after the camera 11 is kept still for a predetermined time (for example, 10 seconds) as the initial value of the stabilization center.

[0186] Next, the user rotates camera 11 in the roll direction a predetermined number of times (e.g., five times) using rotation mechanism 13. While camera 11 is rotating, detection unit 155 detects the position of feature point PF in the image coordinate system of the cut-out image of each frame obtained by camera 11 capturing images in multiple postures with different angles in the roll direction. Detection unit 155 detects the center and radius of the trajectory of feature point PF for each rotation as the stabilization center and the variation in the position of feature point PF, respectively.

[0187] Any method can be used to detect the center and radius of the locus of the feature points PF. For example, the detection unit 155 uses a Hough transform to detect a circle that is the locus of the feature points PF, and detects the center and radius of the detected circle.

[0188] At this time, if there is a deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR of the rotation mechanism 13, the detection unit 155 corrects the detection result of the stabilization center using the measured distance L measured by the distance measuring sensor 15, for example, by the method described with reference to Figures 14 to 16. That is, the detection unit 155 calculates the deviation between the center of the camera coordinate system of the camera 11 and the rotation axis AR of the rotation mechanism 13 using the measured distance L measured by the distance measuring sensor 15, and corrects the detection result of the stabilization center based on the calculated deviation amount.

[0189] The detection unit 155 calculates the average of the stabilization centers and the average of the variations in the positions of the feature points PF.

[0190] At this time, if there is a misalignment between the center of the camera coordinate system of the camera 11 and the rotation axis AR of the rotation mechanism 13, the detection unit 155 may correct the average of the stabilization centers, for example, by the method described with reference to Figures 14 to 16.

[0191] The detection unit 155 calculates the difference between the initial value of the stabilization center and the average of the stabilization centers as the amount of deviation of the stabilization center.

[0192] At this time, if there is a misalignment between the center of the camera coordinate system of the camera 11 and the rotation axis AR of the rotation mechanism 13, the detection unit 155 may correct the amount of misalignment of the stabilization center, for example, by the method described with reference to Figures 14 to 16.

[0193] The detection unit 155 calculates the principal point of the camera 11 by subtracting the amount of deviation of the stabilization center from the currently set principal point of the camera 11 .

[0194] The calculated coordinates of the principal point of the camera 11 are the coordinates obtained by converting the stabilization center in the image coordinate system of the cut-out image into the stabilization center in the image coordinate system of the captured image.

[0195] In step S12, the detection unit 155 determines whether or not to end the detection process of the stabilization center. If it is determined that the detection process of the stabilization center is to be continued, the process proceeds to step S13.

[0196] In step S13, the measurement system 1 updates the principal point of the camera 11 used in the stabilization process.

[0197] Specifically, the detection unit 155 supplies principal point information indicating the principal point of the camera 11 calculated in the processing of step S11 to the mesh creation unit 153, and also transmits it to the camera 11 via the I / O interface unit 151.

[0198] In response to this, the parameter setting unit 121 of the camera 11 receives the principal point information from the external terminal 12 via the I / O interface unit 128. The parameter setting unit 121 updates the currently set principal point of the camera 11 to a value indicated in the principal point information. The parameter setting unit 121 supplies parameter information relating to internal parameters including the updated principal point of the camera 11 to the stabilization processing unit 124.

[0199] Thereafter, the process returns to step S1, and steps S1 to S13 are repeatedly executed until it is determined in step S12 that the stabilization center detection process is terminated. As a result, in the next detection process, the measurement distance L to the measurement object 14 is measured based on the detection result of the stabilization center, and the stabilization center detection process is repeatedly executed while updating the principal point of the camera 11 used in the stabilization process. That is, the measurement distance L to the measurement object 14 is measured every time the stabilization center detection process is repeated.

[0200] On the other hand, if it is determined in step S12 that the stabilization center detection process is to be ended, the process proceeds to step S14. For example, when the detection unit 155 has executed the stabilization center detection process a predetermined number of times (for example, five times), it determines that the stabilization center detection process is to be ended.

[0201] In step S14, the measurement system 1 corrects the principal point of the camera 11.

[0202] For example, the detection unit 155 of the external terminal 12 executes the detection process of the stabilization center multiple times, and determines the principal point of the camera 11 to be the principal point of the camera 11 that was used for the stabilization process in the detection process in which the average variation in the position of the feature point PF was the smallest.

[0203] The detection unit 155 supplies principal point information indicating the set principal point of the camera 11 to the mesh creation unit 153 , and also transmits the same to the camera 11 via the I / O interface unit 151 .

[0204] In response to this, the parameter setting unit 121 of the camera 11 receives the principal point information from the external terminal 12 via the I / O interface unit 128. The parameter setting unit 121 corrects the currently set principal point of the camera 11 to a value indicated in the principal point information. The parameter setting unit 121 supplies parameter information relating to internal parameters including the corrected principal point of the camera 11 to the stabilization processing unit 124.

[0205] In step S15, the measurement system 1 corrects the camera coordinate system mesh.

[0206] Specifically, the mesh creation unit 153 of the external terminal 12 creates a mesh of the camera coordinate system based on the internal parameters including the principal point of the camera 11 corrected in the processing of step S14. The mesh creation unit 153 generates mesh information on the created mesh of the camera coordinate system and transmits it to the camera 11 via the I / O interface unit 151.

[0207] The mesh setting unit 122 of the camera 11 receives mesh information from the external terminal 12 via the I / O interface unit 128. The mesh setting unit 122 corrects the mesh in the camera coordinate system based on the received information. The mesh setting unit 122 supplies mesh information on the corrected mesh to the image cropping unit 127.

[0208] Then, the calibration process ends.

[0209] After the calibration process is completed, the user checks the operation of the image stabilization function. Specifically, the user turns on / off the power supply of the camera 11, sets the image stabilization function to on, and then rotates the rotation mechanism 13 to check that the feature points of the measurement object 14 are stationary and do not move.

[0210] In particular, in the present technology, for example, while a user rotates the camera 11 in the roll direction by hand without using the rotation mechanism 13, the detection unit 155 can detect the stabilization center by the calibration process described with reference to FIG. 22.

[0211] That is, even if the camera 11 moves back and forth due to the movement of the user's hand, the accurate distance between the camera and the measurement object is measured, so that the stabilization center can be detected with high accuracy. In this case, as shown in Fig. 25, the locus TR of the feature points PF in the cropped image when the camera 11 is rotated does not necessarily have to be circular, and may be, for example, elliptical or irregular. Then, similar to the case where the locus TR of the feature points PF is circular, the center of the locus TR of the feature points PF may be detected as the stabilization center.

[0212] As a result, each time the principal point of the camera 11 used in the stabilization process is updated, the distorted shape of the locus TR of the feature point PF in the cropped image becomes smaller, as shown in Figure 26, and eventually converges to the feature point PF.

[0213] In this case, similarly to the above-mentioned method, the cropped image, the feature points, the trajectory of the feature points, and the like are displayed on the display operation unit 156 of the external terminal 12. Therefore, the user can manually rotate the camera 11 in the roll direction while checking the captured image and the state of the feature points, and detect the stabilization center.

[0214] In this case, the rotation mechanism 13 is not necessary, so that the stabilization center can be detected more easily.

[0215] According to the above processing, a distance measuring sensor 15 is newly provided and linked to the camera 11 to measure the distance (measurement distance L) between the camera 11 and the object to be measured 14 each time a photograph is taken, thereby deriving the central value of the stabilization processing.

[0216] This allows the accurate distance between the camera and the object to be measured to be measured even if the camera 11 moves back and forth due to the movement of the user's hand, making it possible to further improve the accuracy of detection of the stabilization center.

[0217] For example, in a lens-interchangeable camera, the user can perform accurate calibration more easily without having to measure the distance between the camera and the object to be measured every time the lens is changed, which in turn makes it possible to expand the market for new cameras that did not exist before.

[0218] <5.Other> (Modification) For example, in step S12 of Fig. 22, when the average of the variation in the positions of the feature points PF becomes equal to or less than a predetermined threshold, the detection process of the stabilization center may be terminated. That is, the detection process of the stabilization center may be repeated until the average of the variation in the positions of the feature points PF becomes equal to or less than a predetermined threshold. Then, the principal point of the camera 11 used in the detection process in which the average of the variation in the positions of the feature points PF becomes equal to or less than the predetermined threshold may be determined as the principal point of the camera 11.

[0219] For example, instead of the detection unit 155, the parameter setting unit 121 may set the principal point of the camera 11 based on the detection result of the stabilization center.

[0220] Further, in this technology, other modified examples disclosed in Patent Document 2 can also be applied.

[0221] (Examples of application of this technology) This technology can be applied to various cameras equipped with a motion sensor and image stabilization function (e.g., digital still cameras, digital video cameras, security cameras, mobile cameras, etc.), and to devices and systems that adjust the principal points of those cameras. This technology can also be applied to components (e.g., LSIs, etc.) and software that process image stabilization functions using motion sensors.

[0222] (Example of computer configuration) The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0223] FIG. 27 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0224] In a computer 1000, a central processing unit (CPU) 1001, a read only memory (ROM) 1002, and a random access memory (RAM) 1003 are interconnected by a bus 1004.

[0225] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0226] The input unit 1006 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0227] In the computer 1000 configured as above, the CPU 1001 performs the above-mentioned series of processes by, for example, loading a program recorded in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0228] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0229] In the computer 1000, the program can be installed in the storage unit 1008 via the input / output interface 1005 by mounting the removable medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0230] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0231] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0232] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0233] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0234] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0235] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0236] (Example of using the system to set the principal point of the internal parameters) FIG. 28 is a diagram showing an example of the use of the system for setting the principal point of the internal parameters described above.

[0237] The above-described system can be used in various cases for sensing light, such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows:

[0238] -Digital cameras, mobile devices with camera functions, and other devices that take images for viewing - Devices used for traffic purposes, such as on-board sensors that take pictures of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping and for recognition of the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliances according to those gestures. -Medical and healthcare equipment, such as endoscopes and devices that take blood vessel images by receiving infrared light - Equipment used for security purposes, such as surveillance cameras for crime prevention and cameras for person recognition - Cosmetic devices such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp -Sports equipment, such as action cameras and wearable cameras for sports use Agricultural equipment, such as cameras for monitoring fields and crop conditions

[0239] The present technology can also be configured as follows. (1) a detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of feature points in a plurality of captured images obtained by capturing images of a measurement object in a plurality of postures with different roll angle angles while the camera is performing a stabilization process, and based on a measured distance between the camera and the measurement object measured by a distance measuring sensor; An information processing device. (2) The detection unit is detecting, as the stabilization center, a center of a trajectory of the feature points in the captured image captured while the camera is rotated in a roll direction by a user's hand in a state in which the camera is executing the stabilization process; The measurement distance is used to correct the detection result of the stabilization center. An information processing device as described in (1). (3) The detection unit is Using the measured distance, a deviation between a rotation axis in a roll direction of the camera and a center of a camera coordinate system of the camera is calculated; The detection result of the stabilization center is corrected based on the amount of deviation. An information processing device according to (2). (4) The measurement distance is measured each time the detection process of the stabilization center is repeated. An information processing device according to any one of (1) to (3). (5) The measured distance is measured by the distance measuring sensor which rotates in a roll direction together with the camera. An information processing device according to any one of (1) to (4). (6) The detection unit detects feature points of the measurement object that indicate a predetermined pattern in the captured image. An information processing device according to any one of (1) to (5). (7) The measurement object includes a single feature point. (6) An information processing device. (8) The measurement object includes a plurality of the feature points. (6) An information processing device. (9) At least a part of the object to be measured is formed with an uneven shape or holes. An information processing device according to any one of (6) to (8). (10) a UI processing unit that displays information indicating the center of the captured image and the positions of the feature points on the captured image by superimposing the information on the captured image. An information processing device according to any one of (1) to (9). (11) A detection process of a stabilization center, which is the center of the stabilization process, is performed based on the positions of feature points in a plurality of captured images obtained by capturing images of a measurement object in a plurality of postures with different roll direction angles while the camera is performing a stabilization process, and the measured distance between the camera and the measurement object measured by a distance measuring sensor. Information processing methods. (12) A camera and A distance measuring sensor; a detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of feature points in a plurality of captured images obtained by capturing images of a measurement object in a plurality of postures with different roll direction angles while the camera is performing a stabilization process, and based on a measured distance between the camera and the measurement object measured by the distance measuring sensor; and A measurement system comprising: (13) The distance measuring sensor is externally attached to the camera. (12) A measurement system according to (12). (14) The distance measuring sensor is built into the camera. (12) A measurement system according to (12). (15) The distance measuring sensor is integrated into the image sensor of the camera. (12) A measurement system according to (12). (16) A rotation mechanism for rotating the camera in a roll direction around a rotation axis is further provided, The distance measuring sensor is rotated together with the camera by the rotation mechanism. A measurement system according to any one of (12) to (15). (17) The rotation mechanism includes a rotation jig for rotating the camera and the distance measuring sensor. (16) A measurement system according to (16). (18) The rotation mechanism further includes an angle jig capable of changing angles of the camera and the distance measuring sensor in a pitch direction and a yaw direction. (17) A measuring system according to (17). [Explanation of symbols]

[0240] 1 Measurement system, 11 Camera, 12 External terminal, 13 Rotation mechanism, 14 Measurement object, 15 Distance measurement sensor, 103 Lens, 104 Image sensor, 105 Motion sensor, 106 DSP, 107 Stabilization chip, 121 Parameter setting section, 122 Mesh setting section, 123 Offset setting section, 124 Stabilization processing section, 125 UI processing section, 126 Image cropping section, 152 UI processing section, 153 Mesh creation section, 154 Offset adjustment section, 155 Detection section, 156 Display operation section, 181 Distance measurement processing section

Claims

1. A detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of feature points in a plurality of captured images obtained by capturing images of a measurement object in a plurality of postures with different roll direction angles while the camera is performing a stabilization process, and a measured distance between the camera and the measurement object measured by a distance measuring sensor. An information processing device comprising:

2. The detection unit is detecting, as the stabilization center, a center of a trajectory of the feature points in the captured image captured while the camera is rotated in a roll direction by a user's hand in a state in which the camera is executing the stabilization process; The measurement distance is used to correct the detection result of the stabilization center. The information processing device according to claim 1 .

3. The detection unit is Using the measured distance, a deviation between a rotation axis in a roll direction of the camera and a center of a camera coordinate system of the camera is calculated; The detection result of the stabilization center is corrected based on the amount of deviation. The information processing device according to claim 2 .

4. The measurement distance is measured each time the detection process of the stabilization center is repeated. The information processing device according to claim 1 .

5. The measured distance is measured by the distance measuring sensor which rotates in a roll direction together with the camera. The information processing device according to claim 1 .

6. The detection unit detects feature points of the measurement object that indicate a predetermined pattern in the captured image. The information processing device according to claim 1 .

7. The measurement object includes a single feature point. The information processing device according to claim 6.

8. The measurement object includes a plurality of the feature points. The information processing device according to claim 6.

9. At least a part of the object to be measured is formed with an uneven shape or holes. The information processing device according to claim 6.

10. The image capturing apparatus further includes a UI processing unit that displays information indicating a center of the captured image and a position of the feature point on the captured image in a superimposed manner. The information processing device according to claim 1 .

11. A detection process of a stabilization center, which is the center of the stabilization process, is performed based on the positions of feature points in a plurality of captured images obtained by capturing images of a measurement object in a plurality of postures with different roll direction angles while the camera is performing a stabilization process, and the measured distance between the camera and the measurement object measured by a distance measuring sensor. Information processing methods.

12. Camera and A distance measuring sensor; a detection unit that detects a stabilization center, which is the center of the stabilization process, based on positions of feature points in a plurality of captured images obtained by capturing images of a measurement object in a plurality of postures with different roll direction angles while the camera is performing a stabilization process, and based on a measured distance between the camera and the measurement object measured by the distance measuring sensor; and A measurement system comprising:

13. The distance measuring sensor is externally attached to the camera.

13. The measurement system of claim 12.

14. The distance measuring sensor is built into the camera.

13. The measurement system of claim 12.

15. The distance measuring sensor is integrated into the image sensor of the camera.

13. The measurement system of claim 12.

16. A rotation mechanism for rotating the camera in a roll direction around a rotation axis is further provided, The distance measuring sensor is rotated together with the camera by the rotation mechanism.

13. The measurement system of claim 12.

17. The rotation mechanism includes a rotation jig for rotating the camera and the distance measuring sensor.

17. The measurement system of claim 16.

18. The rotation mechanism further includes an angle jig capable of changing angles of the camera and the distance measuring sensor in a pitch direction and a yaw direction.

18. The measurement system of claim 17.

Images