Information Processing Apparatus, Information Processing Method, and Information Processing System

Abstract

The present disclosure relates to an information processing device, an information processing method, and an information processing system capable of achieving simpler camera calibration. A control unit generates a first trigger for displaying a pattern imaged by a camera on a mobile terminal in a case where calibration of the camera of a mobile unit is required, and generates a second trigger for presenting feedback to a user based on whether the camera is likely to appropriately image the pattern. For example, the technology according to the present disclosure can be applied to a mobile unit such as a drone or a mobile terminal such as a smartphone.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing system, and more particularly to an information processing device, an information processing method, and an information processing system capable of achieving simpler camera calibration. Background Art

[0002] In recent years, various methods have been proposed to calibrate cameras onboard drones.

[0003] For example, Patent Document 1 discloses a calibration method in which a calibration signal is generated based on posture information obtained from an image captured by an imaging device, and a guide signal is displayed on a screen together with the calibration signal.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: U.S. Patent Application Publication No. 2019-0156517 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] Conventionally, calibration of a camera mounted on a drone is accomplished by displaying a calibration pattern, such as a circular grid, on a monitor while the user performs imaging with the drone in his / her hands.

[0009] However, with this method, a monitor for display and a personal computer (PC) for operation must be prepared, and the user must operate the PC while holding the drone in his / her hand for imaging. In addition, it is not easy for the user to determine whether imaging is performed appropriately.

[0010] The present disclosure has been completed in view of such circumstances, and an object thereof is to achieve simpler camera calibration.

[0011] Solution to the problem

[0012] The information processing device disclosed in the present invention is an information processing device including a control unit, which generates a first trigger for displaying a pattern imaged by the camera on a mobile terminal when calibration of the camera of the mobile unit is required, wherein the control unit generates a second trigger for presenting feedback to the user based on whether the camera is likely to properly image the pattern.

[0013] The information processing method disclosed herein is an information processing method, comprising generating, by an information processing device, a first trigger for displaying a pattern imaged by a camera on a mobile terminal when calibration of the camera of a mobile unit is required, and generating a second trigger for presenting feedback to a user based on whether the camera is likely to properly image the pattern.

[0014] The information processing system disclosed in the present invention is an information processing system, comprising: a mobile unit; a mobile terminal; a first control unit, which generates a first trigger for displaying a pattern imaged by the camera on the mobile terminal when calibration of the camera of the mobile unit is required; and a second control unit, which generates a second trigger for presenting feedback to the user based on whether the camera can properly image the pattern.

[0015] In the present disclosure, in the case where calibration of the camera of the mobile unit is required, a first trigger is generated for displaying a pattern imaged by the camera on the mobile terminal, and a second trigger is generated for presenting feedback to the user based on whether the camera is likely to properly image the pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram illustrating an overview of a camera calibration system to which the technology according to the present disclosure is applied.

[0017] Figure 2 is a block diagram illustrating an example of a functional configuration of a mobile unit.

[0018] Figure 3 is a block diagram illustrating a functional configuration example of a mobile terminal.

[0019] Figure 4 is a flow chart illustrating an example of camera calibration.

[0020] Figure 5 is a flow chart illustrating an example of camera calibration.

[0021] Figure 6 is a diagram illustrating an example of a calibration pattern.

[0022] Figure 7 is a diagram illustrating a display example of a calibration pattern.

[0023] Figure 8 is a diagram illustrating a display example of a calibration pattern.

[0024] Fig. 9 is a diagram illustrating a display example of a calibration pattern.

[0025] Fig.10 is a diagram illustrating a display example of a calibration pattern.

[0026] Fig.11is a diagram illustrating a display example of a calibration pattern.

[0027] Fig.12 is a flow chart illustrating another example of camera calibration.

[0028] Fig.13 is a flow chart illustrating another example of camera calibration.

[0029] Fig.14 is a block diagram illustrating a configuration example of a computer. DETAILED DESCRIPTION

[0030] Hereinafter, a mode for carrying out the present disclosure (hereinafter referred to as an embodiment) will be described. Note that the description will be given in the following order.

[0031] 1. Configuration example of camera calibration system

[0032] 2. Example of Camera Calibration (Example of Determination Processing on the Mobile Unit Side)

[0033] 3. Example of calibration pattern display

[0034] 4. Another Example of Camera Calibration (Example of Determination Processing on the Mobile Terminal Side)

[0035] 5. Computer configuration example

[0036] <1. Configuration example of camera calibration system>

[0037] (Overview of the camera calibration system)

[0038] Figure 1 is a diagram illustrating an overview of a camera calibration system configured as an information processing system to which the technology according to the present disclosure is applied.

[0039] Figure 1 The camera calibration system 1 shown in illustrative examples includes a mobile unit 10 , a camera 20 included in the mobile unit 10 , and a mobile terminal 30 .

[0040] The mobile unit 10 is configured as a drone, for example, and moves autonomously (flies autonomously) according to a pre-set flight path or moves (flies) according to a user's operation. In addition to a drone, the mobile unit 10 may also be configured as a robot such as a vehicle, a ship, or a vacuum cleaner.

[0041] Hereinafter, an example in which the technology according to the present disclosure is applied to a drone flying in the air will be described. However, the technology according to the present disclosure may also be applied to robots such as vehicles moving on land, ships moving on or under water, and vacuum cleaners moving indoors.

[0042] A camera 20 that captures images while the mobile unit 10 moves is carried on the mobile unit 10 .

[0043] The camera 20 is connected to the bottom surface of the body of the mobile unit 10 via, for example, an electronically drivable gimbal, and includes a zoom lens. For example, in addition to imaging the front side of the mobile unit 10, the camera 20 can also image the bottom side, left and right directions, rear side, etc. of the mobile unit 10.

[0044] The camera 20 may be a monocular camera or a stereo camera. In addition, the number of cameras 20 mounted on the mobile unit 10 is not limited to one, and a plurality of cameras 20 may be mounted on the mobile unit 10.

[0045] The mobile terminal 30 includes a tablet terminal or a smart phone operated by a user, a portable mobile personal computer (PC), etc. A calibration pattern (hereinafter also simply referred to as a pattern) used for calibration of the camera 20 is displayed on a display unit (display) of the mobile terminal 30. Based on the calibration, camera parameters such as internal parameters indicating a focal length, deviation of an image center, lens distortion, etc., which are specific to the camera 20, and external parameters indicating a position and an attitude of the camera 20 are estimated.

[0046] Conventionally, calibration of a camera mounted on a drone is accomplished by displaying a pattern, such as a circular grid, on a monitor while the user performs imaging with the drone in his / her hands.

[0047] However, with this method, a monitor for display and a PC for operation must be prepared, and the user must operate the PC while holding the drone in his / her hand for imaging. In addition, it is not easy for the user to determine whether imaging is performed appropriately.

[0048] On the other hand, in the camera calibration system 1, the user can perform calibration in a state where the mobile unit 10 is placed on a flat place such as a table. Specifically, in this state, calibration is performed by the user holding a pattern displayed on the mobile terminal 30 over the camera 20, thereby imaging the pattern at various angles. As described above, according to the camera calibration system 1, simpler camera calibration can be achieved.

[0049] (Functional configuration example of mobile unit)

[0050] Figure 2 is a block diagram illustrating a functional configuration example of the mobile unit 10 .

[0051] The mobile unit 10 includes a control unit 51 , a driving mechanism 52 , a communication unit 53 , and a storage unit 54 .

[0052] The control unit 51 includes a processor such as a central processing unit (CPU), a memory, and the like, and controls the drive mechanism 52, the communication unit 53, and the storage unit 54 by executing a predetermined program. In addition, the control unit 51 controls imaging performed by the camera 20. In addition, the control unit 51 generates triggers for executing various types of processing.

[0053] The driving mechanism 52 is a mechanism for moving the mobile unit 10. In this example, the mobile unit 10 is a drone, and the driving mechanism 52 includes a motor, a propeller, and the like as a flying mechanism.

[0054] The communication unit 53 includes a network interface, etc., and performs wireless or wired communication with the mobile terminal 30, a controller for operating the mobile unit 10, etc. For example, the communication unit 53 may communicate directly with a communication partner device, or may perform network communication via Wi-Fi (registered trademark), a 4G or 5G base station, or a repeater.

[0055] The storage unit 54 includes a nonvolatile memory such as a flash memory, and stores various types of information under the control of the control unit 51 .

[0056] (Functional configuration example of mobile terminal)

[0057] Figure 3 is a block diagram illustrating a functional configuration example of the mobile terminal 30 .

[0058] The mobile terminal 30 includes a control unit 71 , a presentation unit 72 , a communication unit 73 , a storage unit 74 , and a sensor 75 .

[0059] The control unit 71 includes a processor such as a CPU, a memory, and the like, and controls the presentation unit 72 , the communication unit 73 , the storage unit 74 , and the sensor 75 by executing a predetermined program.

[0060] The presentation unit 72 includes an organic electroluminescence (EL) display, a liquid crystal display, a speaker, a motor that rotates an eccentric weight, a linear resonance actuator, and the like, and presents information to the user through display, sound, vibration, and the like.

[0061] The communication unit 73 includes a network interface and the like, and performs wireless or wired communication with the mobile unit 10 .

[0062] The storage unit 74 includes a nonvolatile memory such as a flash memory, and stores various types of information under the control of the control unit 71 .

[0063] The sensor 75 includes an acceleration sensor, a gyro sensor, and the like, and acquires sensor data such as acceleration data and angular velocity data of the mobile terminal 30 .

[0064] <2. Example of camera calibration>

[0065] Next, we will refer to Figure 4 and Figure 5 An example of camera calibration by the camera calibration system 1 is described.

[0066] Figure 4 and Figure 5 The process in starts when calibration is required in any one of the cameras 20 included in the mobile unit 10, and is performed by the communication unit 53 of the mobile unit 10 and the communication unit 73 of the mobile terminal 30 communicating with each other.

[0067] For example, whether calibration is required in the camera 20 is determined by comparing left and right images captured by the camera 20 configured as a stereo camera.

[0068] In step S11 , the control unit 51 of the mobile unit 10 generates a trigger for presenting guidance for notifying the user that the camera 20 is to be calibrated, thereby instructing the mobile terminal 30 to present the guidance.

[0069] In step S31 , the control unit 71 of the mobile terminal 30 presents guidance based on the instruction from the mobile unit 10 .

[0070] According to the distance between the camera 20 to be calibrated and the mobile terminal 30, the guidance is presented by at least one of vibration or sound in the mobile terminal 30. For example, as the user brings the mobile terminal 30 closer to the camera 20 requiring calibration, the vibration of the mobile terminal 30 increases, the sound output from the mobile terminal 30 increases, or the sound itself changes.

[0071] Furthermore, as a guide, an arrow indicating the direction in which the camera 20 exists may be displayed on the display of the mobile terminal, so that the user can more reliably bring the mobile terminal 30 closer to the camera 20 requiring calibration.

[0072] Also, the guide may be presented by lighting of a lighting unit such as a light emitting diode (LED) light emitter provided in the camera 20 requiring calibration.

[0073] Therefore, the user may move the mobile terminal 30 close to the camera 20 to be calibrated.

[0074] When such guidance is presented, in step S12, the control unit 51 of the mobile unit 10 controls the camera 20 to be calibrated to start capturing images by the camera 20. The captured images may be real-time moving images or still images captured at predetermined time intervals. For example, image capture by the camera 20 is continued until imaging for calibration is started.

[0075] In step S13, the control unit 51 of the mobile unit 10 determines whether the shape of a smartphone (mobile terminal 30) has been detected in the image captured by the camera 20 to be calibrated. This process is repeated until the shape of a smartphone is detected. On the other hand, when the shape of a smartphone is detected, the process proceeds to step S14. According to the process in step S13, it is possible to prevent a part of the pattern displayed on the mobile terminal 30 from being cut off in the image captured by the camera 20.

[0076] In step S14 , the control unit 51 of the mobile unit 10 generates a trigger for displaying a pattern imaged by the camera 20 to be calibrated on the mobile terminal 30 , thereby instructing the mobile terminal 30 to display the pattern.

[0077] In step S32 , the control unit 71 of the mobile terminal 30 displays a pattern on the display constituting the presentation unit 72 based on the instruction from the mobile unit 10 .

[0078] Figure 6 is a diagram illustrating an example of a pattern (calibration pattern) displayed on the mobile terminal 30 .

[0079] Figure 6 Figure A of the drawings illustrates a circular grid of a circular symmetric pattern, while Figure B illustrates a circular grid of a circular asymmetric pattern. In addition, Figure C illustrates a chessboard.

[0080] In the calibration of the camera 20, the camera parameters are based on multiple imaging from different positions and angles such as Figure 6 It is estimated from the image of a pattern with known size shown in .

[0081] In step S15, the control unit 51 of the mobile unit 10 controls the driving of the gimbal of the camera 20, for example, to start guiding the camera 20 to a proper imaging position. The camera 20 may be guided to a proper imaging position by the movement of the mobile unit 10 itself.

[0082] Then, in step S16, the control unit 51 of the mobile unit 10 determines whether appropriate imaging (imaging for calibration) is possible based on the image captured by the camera 20. Steps S15 and S16 are repeated until appropriate imaging becomes possible, and when appropriate imaging becomes possible, the process proceeds to step S17. Whether appropriate imaging is possible is determined, for example, by a pattern appearing in the image captured by the camera 20.

[0083] In step S17 , the control unit 51 of the mobile unit 10 generates a trigger for presenting feedback indicating that appropriate imaging has become possible to the user, thereby instructing the mobile terminal 30 to present the feedback.

[0084] In step S33 , the control unit 71 of the mobile terminal 30 presents feedback based on the instruction from the mobile unit 10 .

[0085] The feedback is presented by at least one of vibration or sound in the mobile terminal 30. For example, when the user holds the mobile terminal 30 above the camera 20 or the like that needs calibration, and the mobile terminal 30 (pattern) is arranged at a position where appropriate imaging is possible, the mobile terminal 30 vibrates or a sound is output from the mobile terminal 30.

[0086] When presenting feedback to the user, Figure 5 In step 34, the control unit 71 of the mobile terminal 30 controls the sensor 75 to acquire sensor data such as acceleration data and angular velocity data of the mobile terminal 30. The acquired sensor data is transmitted to the mobile unit 10.

[0087] In step S18, the control unit 51 of the mobile unit 10 controls imaging for calibration of the camera 20 according to the sensor data from the mobile terminal 30. For example, the control unit 51 of the mobile unit 10 sets at least one of the shutter speed of the camera 20 or the aperture value of the lens based on the sensor data acquired in the mobile terminal 30.

[0088] For example, in the case where the user holds the mobile terminal 30 (pattern) above the camera 20 to relatively slowly move the mobile terminal 30, the shutter speed is set to be low according to the acceleration data and angular velocity data of the mobile terminal 30 at this time. On the contrary, in the case where the user holds the mobile terminal 30 above the camera 20 to relatively quickly move the mobile terminal, the shutter speed is set to be high according to the acceleration data and angular velocity data of the mobile terminal 30 at this time.

[0089] By imaging the pattern by exposing with the shutter speed and aperture value set in this way, the accuracy of calibration can be improved.

[0090] In step S19, the control unit 51 of the mobile unit 10 determines whether the number of captured images obtained by imaging for calibration is sufficient. If it is determined that the number of captured images is insufficient, steps S18 and S19 are repeated, and if it is determined that the number of captured images is sufficient, the process proceeds to step S20.

[0091] Note that feedback indicating that proper imaging has become possible may be presented for each imaging for calibration.

[0092] In step S20 , the control unit 51 of the mobile unit 10 performs calibration based on images obtained through a plurality of imagings to estimate camera parameters.

[0093] Thereafter, in step S21 , the control unit 51 of the mobile unit 10 images the pattern by controlling the camera 20 , corrects the obtained image using the estimated camera parameters, and then checks the corrected image.

[0094] In step S22, the control unit 51 of the mobile unit 10 determines whether the correction of the checked image is sufficient. If it is determined that the correction is insufficient, the estimated camera parameters may be inappropriate. Therefore, the process returns to step S18 and the imaging, calibration and image checking are repeated.

[0095] On the other hand, if it is determined that the correction is sufficient, that is, if the estimated camera parameters are appropriate, the process proceeds to step S23.

[0096] In step S23 , the control unit 51 of the mobile unit 10 generates a trigger for presenting feedback to the user indicating that the calibration has been completed correctly, thereby instructing the mobile terminal 30 to present the feedback.

[0097] In step S35 , the control unit 71 of the mobile terminal 30 presents feedback based on the instruction from the mobile unit 10 .

[0098] The feedback here is presented by at least one of sound or display in the mobile terminal 30. For example, a sound indicating that the calibration has been completed correctly is output from the mobile terminal 30, or a screen indicating that the calibration has been completed correctly is displayed on the display constituting the presentation unit 72.

[0099] According to the above-described process, calibration is performed by the user holding a pattern displayed on the mobile terminal 30 over the camera 20 to image the pattern at various angles, and thus simpler camera calibration can be achieved.

[0100] Furthermore, since feedback to the user is presented according to whether appropriate imaging is possible, the user can easily determine that appropriate imaging is performed.

[0101] Moreover, since the user simply holds the mobile terminal 30 in his / her hand and holds the mobile terminal 30 above the camera 20, the user's burden can be reduced and the sense of operation can be improved compared to the case where the drone performs imaging in the user's hand.

[0102] <3. Display example of calibration pattern>

[0103] Here, a display example of the calibration pattern will be described.

[0104] (index)

[0105] In calibration, an index is assigned to each figure having a known size arranged in the calibration pattern. At this time, in a case where a part of the pattern displayed on the mobile terminal 30 is cut off or covered by the user's finger or the like, the index is not correctly assigned to the figure, or the assigned index is not referred to. Therefore, there is a possibility that appropriate calibration cannot be performed.

[0106] So, for example, Figure 7 As shown in , a pattern 110 in which figures of different sizes are periodically arranged is displayed. In the pattern 110, circles of three sizes are periodically arranged. According to the pattern 110, when a part of the pattern 110 is cut off or covered by the user's finger or the like, it is possible to detect that the circles of three sizes are not periodically arranged. In this case, by presenting an alert to the user and appropriately displaying the pattern 110, appropriate calibration can be performed.

[0107] Similarly, if Figure 8 As shown in , a pattern 120 in which figures having different shapes are periodically arranged can be displayed. In the pattern 120, a circle, a square, and a triangle are periodically arranged. According to the pattern 120, when a part of the pattern 120 is cut off or covered by a user's finger or the like, it is possible to detect that the figures having three shapes are not periodically arranged. In this case, by presenting a warning to the user and appropriately displaying the pattern 120, appropriate calibration can also be performed.

[0108] (Overall shape of the calibration pattern)

[0109] like Figure 6 It is also possible to change the shape of the entire pattern in which figures are periodically arranged, as in the pattern shown in FIG.

[0110] Fig. 9 is a diagram illustrating a display example of a calibration pattern whose overall shape is changed. Fig. 9 An example in which the shape of the entire circular mesh having a circular symmetrical pattern is changed on the mobile terminal 30 is illustrated.

[0111] Specifically, Fig. 9 FIG. A shows an isosceles trapezoidal circular grid 130a with the shorter bottom side of the trapezoid located at the upper side. The circular grid 130a adopts the following display mode: the vertically long rectangular circular grid is tilted with the upper side of the mobile terminal 30 facing the far side of the screen and the lower side of the mobile terminal 30 facing the near side of the screen.

[0112] Fig. 9 FIG. B illustrates an isosceles trapezoidal circular grid 130b with the shorter base of the trapezoid on the left. The circular grid 130b adopts the following display mode: the vertically long rectangular circular grid is tilted with the left side of the mobile terminal 30 toward the far side of the screen and the right side of the mobile terminal 30 toward the near side of the screen.

[0113] Fig. 9 FIG. C illustrates an isosceles trapezoidal circular grid 130c with the shorter base of the trapezoid on the right. The circular grid 130c adopts the following display mode: the vertically long rectangular circular grid is tilted with the right side of the mobile terminal 30 toward the far side of the screen and the left side of the mobile terminal 30 toward the near side of the screen.

[0114] The trigger for changing the shape of the entire pattern as described above is generated by the control unit 51 of the mobile unit 10, and the mobile terminal 30 is instructed to change the shape of the entire pattern.

[0115] By changing the shape of the entire pattern on the mobile terminal 30 in this way, it is possible to display the pattern as if the mobile terminal 30 is tilted at various angles. Therefore, calibration can be achieved even in a state where the mobile terminal 30 is fixed.

[0116] Note that although Fig. 9 In the example of FIG. 3 , the overall shape of the circular grid having a circular symmetrical pattern is changed on the mobile terminal 30 , but it is also possible to change the overall shape of other patterns.

[0117] (Overall size of calibration pattern)

[0118] The size of the entire pattern may be changed according to the distance between the camera 20 and the mobile terminal 30 .

[0119] Fig.10 is a diagram illustrating a display example of a calibration pattern whose overall size is changed according to the distance from the camera 20 . Fig.10 An example in which the size of the entire circular mesh having a circular symmetric pattern is changed on the mobile terminal 30 is illustrated.

[0120] Specifically, on the mobile terminal 30 , the farther (larger) the distance from the camera 20 , the larger the circular grid 140 is displayed, and the closer (smaller) the distance from the camera 20 , the smaller the circular grid 140 is displayed.

[0121] The trigger for changing the size of the entire pattern as described above is generated by the control unit 51 of the mobile unit 10, and the mobile terminal 30 is instructed to change the size of the entire pattern.

[0122] In this manner, by changing the size of the pattern as a whole according to the distance from the camera 20 , the camera 20 can image the pattern of the same size regardless of the distance to the mobile terminal 30 .

[0123] Note that although Fig.10 In the example of FIG. 3 , the size of the entire circular grid having a circularly symmetrical pattern is changed on the mobile terminal 30 , but it is also possible to change the size of the entire other patterns.

[0124] (Color of the entire calibration pattern)

[0125] The color of the overall pattern can be changed.

[0126] Fig.11 is a diagram illustrating a display example of a calibration pattern in which the overall color is changed. Fig.11 An example in which the color of the entire circular mesh having a circular symmetrical pattern is changed on the mobile terminal 30 is illustrated.

[0127] exist Fig.11 In the example of FIG. 1 , a circular grid 150 r in which red circles are arranged is first displayed on the mobile terminal 30 , then a circular grid 150 g in which green circles are arranged is displayed, and finally a circular grid 150 b in which blue circles are arranged is displayed.

[0128] The trigger for changing the color of the entire pattern as described above is generated by the control unit 51 of the mobile unit 10, and the mobile terminal 30 is instructed to change the color of the entire pattern.

[0129] In this way, by changing the color of the pattern as a whole, it is possible to achieve calibration for each of the R, G and B color channels.

[0130] Note that although Fig.11 In the example of FIG. 3 , the color of the entire circular grid having a circularly symmetrical pattern is changed on the mobile terminal 30 , but it is also possible to change the color of the entire other patterns.

[0131] <4. Another Example of Camera Calibration>

[0132] In reference Figure 4 and Figure 5 In the described example of camera calibration, a process is determined on the mobile unit 10 side, such as the control unit 51 of the mobile unit 10 generating a trigger for displaying a pattern or a trigger for presenting feedback.

[0133] Without limitation thereto, in the case where the processing performance of the control unit 71 of the mobile terminal 30 is high, the processing may be determined on the mobile terminal 30 side, for example, by generating a trigger for displaying a pattern or a trigger for presenting feedback through the control unit 71 of the mobile terminal 30 .

[0134] Therefore, reference Fig.12 and Fig.13 , an example of camera calibration for which processing is determined by the mobile terminal 30 will be described.

[0135] Fig.12 and Fig.13The processing in also starts when calibration is required in any one of the cameras 20 included in the mobile unit 10, and is performed by the communication unit 53 of the mobile unit 10 and the communication unit 73 of the mobile terminal 30 communicating with each other.

[0136] In step S111 , the control unit 51 of the mobile unit 10 generates a trigger for presenting guidance for notifying the user that the camera 20 is to be calibrated, thereby instructing the mobile terminal 30 to present the guidance.

[0137] In step S131 , the control unit 71 of the mobile terminal 30 presents guidance based on the instruction from the mobile unit 10 .

[0138] When the guidance is presented, in step S112, the control unit 51 of the mobile unit 10 controls the camera 20 to be calibrated to start capturing images by the camera 20. For example, image capture by the camera 20 is continued until imaging for calibration is started, and the captured images are sequentially transmitted to the mobile unit 10.

[0139] That is, in step S132 , the control unit 71 of the mobile terminal 30 starts acquiring the image transmitted from the mobile unit 10 .

[0140] In step S133, the control unit 71 of the mobile terminal 30 determines whether the shape of the smartphone (mobile terminal 30) has been detected in the image captured by the camera 20 to be calibrated. This process is repeated until the shape of the smartphone is detected. On the other hand, when the shape of the smartphone is detected, the process proceeds to step S134.

[0141] In step S134 , the control unit 71 of the mobile terminal 30 generates a trigger for displaying the pattern imaged by the camera 20 to be calibrated on the mobile terminal 30 , thereby causing the display constituting the presentation unit 72 to start displaying the pattern.

[0142] Thereafter, in step S135, the control unit 71 of the mobile terminal 30 instructs the mobile unit 10 to start guiding the camera 20 to a suitable imaging position. In step S113, the control unit 51 of the mobile unit 10 controls the driving of the gimbal of the camera 20, for example, to start guiding the camera 20 to a suitable imaging position.

[0143] Then, in step S136, the control unit 71 of the mobile terminal 30 determines whether appropriate imaging (imaging for calibration) is possible based on the image sent from the mobile unit 10. Step S136 is repeated until appropriate imaging becomes possible, and when appropriate imaging becomes possible, the process proceeds to step S137.

[0144] In step S137 , the control unit 71 of the mobile terminal 30 generates a trigger for presenting feedback indicating that appropriate imaging becomes possible to the user, thereby causing the presentation unit 72 to present the feedback.

[0145] When presenting feedback to the user, Fig.13 In step 138, the control unit 71 of the mobile terminal 30 controls the sensor 75 to acquire sensor data such as acceleration data and angular velocity data of the mobile terminal 30. The acquired sensor data is transmitted to the mobile unit 10.

[0146] In step S114, the control unit 51 of the mobile unit 10 controls imaging for calibration of the camera 20 according to the sensor data from the mobile terminal 30. In step S115, the control unit 51 of the mobile unit 10 transmits an image obtained by imaging for calibration to the mobile terminal 30.

[0147] In step S139, the control unit 71 of the mobile terminal 30 determines whether the number of captured images transmitted from the mobile unit 10 is sufficient. If the number of captured images is insufficient, steps S138, S114, S115, and S139 are repeated. If it is determined that the number of captured images is sufficient, the process proceeds to step S140.

[0148] In step S140 , the control unit 71 of the mobile terminal 30 performs calibration based on images obtained through a plurality of imagings to estimate camera parameters.

[0149] Thereafter, in step S141 , the control unit 71 of the mobile terminal 30 corrects the image of the pattern imaged therein by the camera 20 using the estimated camera parameters, and then checks the corrected image.

[0150] In step S142, the control unit 71 of the mobile terminal 30 determines whether the correction of the image being checked is sufficient. If it is determined that the correction is insufficient, then the estimated camera parameters may be inappropriate. Therefore, the process returns to step S138 and the imaging, calibration and image checking are repeated.

[0151] On the other hand, if it is determined that the correction is sufficient, that is, if the estimated camera parameters are appropriate, the process proceeds to step S143.

[0152] In step S143 , the control unit 71 of the mobile terminal 30 generates a trigger for presenting feedback indicating that the calibration has been completed correctly to the user, thereby causing the presentation unit 72 to present the feedback.

[0153] Also according to the above-described process, calibration is performed by the user holding a pattern displayed on the mobile terminal 30 over the camera 20 to image the pattern at various angles, and thus simpler camera calibration can be achieved.

[0154] <5. Example of computer configuration>

[0155] The above series of processing can be performed by hardware or software. In the case of performing a series of processing by software, the program included in the software is installed on a computer. Here, the computer includes a computer combined in a dedicated hardware, such as a general-purpose personal computer that can perform various functions by installing various programs, etc.

[0156] Fig.14 : is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processes according to a program.

[0157] The above-mentioned mobile unit 10 and mobile terminal 30 are constituted as information processing devices, and can be operated by a Fig.14 A computer implementation of the configuration shown in .

[0158] In the computer, a CPU 301 , a read only memory (ROM) 302 , and a random access memory (RAM) 303 are connected to each other via a bus 304 .

[0159] An input / output interface 305 is also connected to the bus 304 . An input unit 306 , an output unit 307 , a storage unit 308 , a communication unit 309 , and a drive 510 are connected to the input / output interface 305 .

[0160] The input unit 306 includes a keyboard, a mouse, a microphone, etc. The output unit 307 includes a display, a speaker, etc. The storage unit 308 includes a hard disk, a nonvolatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0161] In the computer configured as described above, for example, the CPU 301 loads the program stored in the storage unit 308 to the RAM 303 through the input / output interface 305 and the bus 304, and executes the above-described series of processing.

[0162] The program executed by the computer (CPU 301) can be provided by being recorded on the removable medium 311 as a package medium, etc. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0163] In the computer, by attaching the removable medium 311 to the drive 310, the program can be installed in the storage unit 308 through the input / output interface 305. In addition, the program can be received by the communication unit 309 through a wired or wireless transmission medium and installed in the storage unit 308. In addition, the program can be pre-installed in the ROM 302 or the storage unit 308.

[0164] Note that the program executed by the computer may be a program that performs processing in time series according to the order described in this specification, or a program that performs processing in parallel or at necessary timing such as when a call is made.

[0165] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present disclosure.

[0166] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all components are in the same housing. Thus, multiple devices housed in separate housings and connected via a network and one device of multiple modules housed in one housing are both systems.

[0167] Furthermore, the effects described in this specification are merely illustrative and not restrictive. Therefore, other effects can be obtained.

[0168] Furthermore, the technology according to the present disclosure may have the following configurations. (1)

[0170] An information processing device comprising

[0171] A control unit generates a first trigger for displaying a pattern imaged by the camera on the mobile terminal when calibration of the camera of the mobile unit is required, wherein

[0172] The control unit generates a second trigger for presenting feedback to the user based on whether the camera is likely to properly image the pattern. (2)

[0174] The information processing device according to (1), wherein

[0175] When the appearance of the mobile terminal is detected, the control unit generates a first trigger. (3)

[0177] The information processing device according to (1) or (2), wherein

[0178] The pattern is a circular grid. (4)

[0180] The information processing device according to (3), wherein

[0181] A circular grid is either a circular symmetrical pattern or a circular asymmetrical pattern. (5)

[0183] The information processing device according to (1) or (2), wherein

[0184] The pattern is a chessboard. (6)

[0186] The information processing device according to (1) or (2), wherein

[0187] The pattern is formed by periodically arranging figures of different sizes. (7)

[0189] The information processing device according to (1) or (2), wherein

[0190] The pattern is formed by periodically arranging figures having different shapes. (8)

[0192] The information processing device according to any one of (1) to (5), wherein

[0193] The control unit changes the shape of the entire pattern in which the figures are periodically arranged. (9)

[0195] The information processing device according to any one of (1) to (5), wherein

[0196] The control unit changes the size of the entire pattern according to the distance between the camera and the mobile terminal. (10)

[0198] The information processing device according to any one of (1) to (5), wherein

[0199] The control unit changes the color of the entire pattern. (11)

[0201] The information processing device according to any one of (1) to (10), wherein

[0202] The feedback is presented through at least one of vibration or sound in the mobile terminal. (12)

[0204] The information processing device according to any one of (1) to (11), wherein

[0205] In case it is possible to properly image the pattern by means of a camera, the control unit controls the imaging by the camera for calibration. (13)

[0207] The information processing device according to (12), wherein

[0208] The control unit sets at least one of a shutter speed of the camera or an aperture value of the lens based on the sensor data acquired in the mobile terminal. (14)

[0210] The information processing device according to (13), wherein

[0211] The sensor data is acquired by at least one of an acceleration sensor or a gyroscope sensor. (15)

[0213] The information processing device according to any one of (1) to (14), wherein

[0214] The control unit generates a third trigger for presenting a guide for notifying the user of the camera that is a target of calibration. (16)

[0216] The information processing device according to (15), wherein

[0217] The guidance is presented through at least one of vibration or sound in the mobile terminal according to the distance between the camera as a target of calibration and the mobile terminal. (17)

[0219] The information processing device according to (15), wherein

[0220] The guide indicates a direction in which a camera as a target of calibration exists on the mobile terminal. (18)

[0222] The information processing device according to (15), wherein

[0223] The guide is presented by lighting of a lighting unit provided on the camera as a target of calibration. (19)

[0225] An information processing method comprising

[0226] By information processing equipment,

[0227] generating a first trigger for displaying a pattern imaged by the camera on the mobile terminal in the event that calibration of the camera of the mobile unit is required, and

[0228] A second trigger for presenting feedback to the user is generated based on whether the camera was likely to properly image the pattern. (20)

[0230] An information processing system, comprising:

[0231] Mobile unit;

[0232] Mobile terminals;

[0233] a first control unit, which generates a first trigger for displaying a pattern imaged by the camera on the mobile terminal when calibration of the camera of the mobile unit is required; and

[0234] A second control unit generates a second trigger for presenting feedback to the user based on whether the camera is likely to properly image the pattern.

[0235] Reference numerals list

[0236] 1 Camera Calibration System

[0237] 10 Mobile Unit

[0238] 20 Camera

[0239] 30 Mobile Terminal

[0240] 51 Control unit

[0241] 52 Driving mechanism

[0242] 53 Communication unit

[0243] 54 storage units

[0244] 71 Control unit

[0245] 72 Presentation Units

[0246] 73 Communication unit

[0247] 74 storage units

[0248] 75 Sensors

Claims

1. An information processing device, comprising a control unit that generates a first trigger for displaying, on a mobile terminal, a pattern imaged by a camera in a case where calibration of the camera of the mobile unit is required, where the control unit generates a second trigger for presenting feedback to a user based on whether the camera is likely to perform proper first imaging of the pattern, wherein in a case where it is possible to perform proper imaging of the pattern by the camera, the control unit controls the first imaging performed by the camera for calibration, the control unit performs calibration based on images obtained by performing the first imaging multiple times from different positions and angles to estimate camera parameters, the control unit controls the camera to perform second imaging of the pattern, corrects the image obtained by the second imaging with the estimated camera parameters, then checks the corrected image, and determines whether the correction of the checked image is sufficient.

2. The information processing device according to claim 1, wherein the control unit generates the first trigger when the form factor of the mobile terminal is detected.

3. The information processing device according to claim 1, wherein the pattern is a circular grid.

4. The information processing device according to claim 3, wherein the circular grid is a circular symmetric pattern or a circular asymmetric pattern.

5. The information processing device according to claim 1, wherein the pattern is a checkerboard.

6. The information processing device according to claim 1, wherein the pattern is formed by periodically arranging graphics of different sizes.

7. The information processing device according to claim 1, wherein the pattern is formed by periodically arranging graphics of different shapes.

8. The information processing device according to claim 1, wherein the control unit changes the overall shape of the pattern in which the graphics are periodically arranged.

9. The information processing device according to claim 1, wherein the control unit changes the overall size of the pattern according to the distance between the camera and the mobile terminal.

10. The information processing device according to claim 1, wherein the control unit changes the overall color of the pattern.

11. The information processing device according to claim 1, wherein the feedback is presented by at least one of vibration or sound in the mobile terminal.

12. The information processing device according to claim 1, wherein the control unit sets at least one of the shutter speed of the camera or the aperture value of the lens based on sensor data acquired in the mobile terminal.

13. The information processing device according to claim 12, wherein the sensor data is acquired by at least one of an acceleration sensor or a gyro sensor.

14. The information processing device according to claim 1, wherein the control unit generates a third trigger for presenting guidance for notifying a user of the camera that is the target of calibration.

15. The information processing device according to claim 14, wherein the guidance is presented by at least one of vibration or sound in the mobile terminal according to the distance between the camera that is the target of calibration and the mobile terminal.

16. The information processing device according to claim 14, wherein the guidance indicates the direction in which the camera that is the target of calibration exists on the mobile terminal.

17. The information processing device according to claim 14, wherein Guide is presented by causing a light-emitting unit provided in a camera that is a target of calibration to emit light.

18. An information processing method, comprising by an information processing device, generating a first trigger for displaying, on a mobile terminal, a pattern imaged by a camera in a case where calibration of the camera of a mobile unit is required, and generating a second trigger for presenting feedback to a user based on whether the camera is likely to perform proper first imaging of the pattern, wherein in a case where it is possible to perform proper imaging of the pattern by the camera, controlling first imaging by the camera for calibration, performing calibration based on images obtained by performing the first imaging multiple times from different positions and angles to estimate camera parameters, controlling the camera to perform second imaging of the pattern, correcting an image obtained by the second imaging with the estimated camera parameters, then checking the corrected image, and determining whether correction of the checked image is sufficient.

19. An information processing system, comprising: a mobile unit; a mobile terminal; a first control unit that generates a first trigger for displaying, on the mobile terminal, a pattern imaged by the camera in a case where calibration of the camera of the mobile unit is required; and a second control unit that generates a second trigger for presenting feedback to the user based on whether the camera is likely to perform proper first imaging of the pattern, wherein, in a case where it is possible to perform proper imaging of the pattern by the camera, the first control unit or the second control unit controls the first imaging by the camera for calibration, the first control unit or the second control unit performs calibration based on images obtained by performing the first imaging multiple times from different positions and angles to estimate camera parameters, the first control unit or the second control unit controls the camera to perform second imaging of the pattern, corrects an image obtained by the second imaging with the estimated camera parameters, then checks the corrected image, and determines whether correction of the checked image is sufficient.

Images