Calibration device, calibration device control method, mobile body, and program

Abstract

This calibration device has: an acquisition unit for acquiring an image of the periphery of a mobile body captured by a camera of the mobile body; a point cloud generation unit for generating point cloud data representing a situation around the mobile body on the basis of the image; a plane detection unit for detecting a plane included in the periphery of the mobile body on the basis of the point cloud data; a change detection unit for detecting that a change has occurred in the mobile body; and a calibration unit for calibrating the camera on the basis of the plane when it is detected that a change has occurred in the mobile body.

Description

Calibration device, calibration device control method, mobile object, and program

[0001] The present invention relates to a calibration device, a control method for a calibration device, a mobile object, and a program.

[0002] A technology has been proposed for recognizing surrounding conditions using point cloud data. When point cloud data is created using images captured by a camera on a mobile body, if the position of the camera changes with respect to the mobile body, the mobile body may not be able to correctly recognize surrounding conditions using the point cloud data. Patent Document 1 proposes a technology for converting image data into three-dimensional coordinate information, estimating the attitude of the camera based on the three-dimensional coordinate information, and calibrating the camera based on the estimation result.

[0003] Japanese Patent Application Laid-Open No. 2020-035158

[0004] Calibrating the camera too frequently increases the processing burden. Some aspects of the present invention provide techniques for calibrating the camera in a timely manner.

[0005] According to some embodiments, there is provided a calibration device comprising: an acquisition unit that acquires an image of the surroundings of a moving body captured by a camera of the moving body; a point cloud generation unit that generates point cloud data representing the situation around the moving body based on the image; a plane detection unit that detects planes contained in the surroundings of the moving body based on the point cloud data; a change detection unit that detects that a change has occurred in the moving body; and a calibration unit that calibrates the camera based on the planes when a change has been detected in the moving body.

[0006] According to some embodiments, the camera can be calibrated in a timely manner.

[0007] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0008] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the invention, and together with the description, serve to explain the principles of the invention. A schematic diagram illustrating an example of the hardware configuration of a mobile body in some embodiments. A schematic diagram illustrating an example of the hardware configuration of a mobile body in some embodiments. A block diagram illustrating an example of the hardware configuration of a control unit of a mobile body in some embodiments. A block diagram illustrating an example of the functional configuration of a control unit of a mobile body in some embodiments. A flow diagram illustrating an example of a driving control method in some embodiments. A flow diagram illustrating an example of a calibration method in some embodiments. A schematic diagram illustrating an example of a calibration method in some embodiments. A schematic diagram illustrating an example of a calibration result in some embodiments.

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] <Configuration of Mobile Body> The configuration of the mobile body 100 will be described with reference to FIGS. 1A and 1B . In the following embodiments, an ultra-compact vehicle with a passenger capacity of approximately one person will be described as an example of the mobile body 100. Such a vehicle may be called micromobility. The vehicle may be an electric vehicle or a vehicle powered by other means. The mobile body 100 recognizes a travel area and generates a route using images captured by the mobile body 100 itself, and autonomously travels according to the generated route. Furthermore, the mobile body 100 generates a route using a positioning result obtained by a positioning sensor and map information. The mobile body 100 may be capable of autonomous travel regardless of whether a person is on board the mobile body 100. The mobile body 100 may be capable of traveling according to the driving operation of an occupant.

[0011] 1A shows a side view of a moving body 100 according to this embodiment, and FIG. 1B shows the internal configuration of the moving body 100. In the figure, arrow X indicates the front-to-rear direction of the moving body 100, with F indicating the front and R indicating the rear. Arrows Y and Z indicate the width direction (left-to-right direction) and up-down direction of the moving body 100, respectively.

[0012] The mobile body 100 is an electric autonomous vehicle that includes a propulsion unit 112 and uses a battery 113 as its main power source. The battery 113 is, for example, a secondary battery such as a lithium-ion battery, and the mobile body 100 is self-propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 takes the form of a tricycle that includes a pair of left and right drive wheels 120 that serve as front wheels, and one driven wheel 121 that serves as a rear wheel. The drive wheels 120 and the driven wheel 121 each include tires. Note that the propulsion unit 112 may take other forms, such as a four-wheeled vehicle. The mobile body 100 includes, for example, a single seat 111. The mobile body 100 includes a luggage rack 123 behind the seat 111. Luggage can be carried on the luggage rack 123.

[0013] The traveling unit 112 includes a drive mechanism 122. The drive mechanism 122 uses motors 122a and 122b as drive sources to rotate the corresponding drive wheels 120. The drive mechanism 122 can move the mobile body 100 forward or backward by rotating each of the drive wheels 120. The drive mechanism 122 can also change the traveling direction of the mobile body 100 by generating a rotation difference between the motors 122a and 122b. The traveling unit 112 includes a driven wheel 121. The driven wheel can rotate around the Z direction as a rotation axis.

[0014] The moving body 100 is equipped with detection units 114 to 116 that detect targets around the moving body 100. The detection units 114 to 116 are a group of sensors that monitor the periphery of the moving body 100. In the present embodiment, the detection units 114 to 116 are all imaging devices (cameras) that capture images of the periphery of the moving body 100, and include, for example, an optical system such as a lens and an image sensor. In the following description, the detection units 114 to 116 are referred to as cameras. Radar or lidar (light detection and ranging) may be used instead of or in addition to the imaging devices.

[0015] The detection units 114 are arranged, for example, in pairs at the front of the moving body 100, spaced apart in the Y direction, and are primarily used to detect targets ahead of the moving body 100. The detection units 115 are arranged on the left and right sides of the moving body 100, respectively, and are primarily used to detect targets to the sides of the moving body 100. The detection unit 116 is arranged at the rear of the moving body 100, and is primarily used to detect targets behind the moving body 100.

[0016] 2 is a block diagram of a control system of the mobile object 100. The mobile object 100 includes a control unit (ECU) 130. The control unit 130 includes one or more processors 131, such as a CPU, a memory device 132, such as a semiconductor memory, an interface with an external device, and the like. Therefore, the control unit 130 is a type of computer. The memory device 132 stores programs executed by the processor, data used for processing by the processor, and the like. A plurality of sets of the processor 131, memory device 132, and interface may be provided for each function of the mobile object 100, and may be configured to be able to communicate with each other.

[0017] The control unit 130 acquires the output (e.g., an image) of the camera, input information from the operation unit 133, audio information input from the audio input device 135, and the like, and executes corresponding processing. The control unit 130 controls the motors 122a and 122b (driving control of the driving unit 112), controls the display of the display panel included in the operation unit 133, and outputs audio alerts and information to the occupants of the moving body 100. The control unit 130 may execute processing using a machine learning model for image recognition on the output from the camera. The control unit 130 may also execute processing using a machine learning model for voice recognition on the output from the audio input device 135. In this manner, the control unit 130 controls the moving body 100. Therefore, the control unit 130 may be considered a control device for the moving body 100. Furthermore, as described below, the control unit 130 calibrates the camera. Therefore, the control unit 130 may be considered a calibration device for the camera of the moving body 100.

[0018] The audio output device 134 includes, for example, a speaker and outputs audio to the occupants of the mobile body 100. The audio input device 135 includes, for example, a microphone and collects audio from the occupants of the mobile body 100. The control unit 130 can recognize the input audio and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 136 is a positioning sensor that receives GNSS signals and detects the current position of the mobile body 100.

[0019] The storage device 137 includes a recording medium for storing various data. The storage device 137 may also store programs executed by the processor, data used by the processor for processing, etc. The storage device 137 may store various parameters of machine learning models for voice recognition and image recognition executed by the control unit 130. The storage device 137 may also store map information of the location where the mobile object 100 travels. The communication device 138 is a communication device capable of communicating with an external device (e.g., a communication terminal 150 owned by a user) via wireless communication such as Wi-Fi (registered trademark) or fifth-generation mobile communication.

[0020] The acceleration sensor 139 is a sensor that measures the acceleration of the mobile object 100. When the mobile object 100 collides with an external object, the acceleration of the mobile object 100 changes. Therefore, the acceleration sensor 139 can be used to measure the impact received by the mobile object 100. The magnitude of the acceleration measured by the acceleration sensor 139 represents the amount of impact received by the mobile object 100. The acceleration sensor 139 transmits the measured value of the acceleration of the mobile object 100 to the control unit 130 wirelessly or via a wired connection.

[0021] The load amount sensor 140 is a sensor that measures the amount of luggage loaded on the loading platform 123. For example, the load amount sensor 140 may measure the weight of the luggage loaded on the loading platform 123. The load amount sensor 140 transmits the measurement value of the luggage amount to the control unit 130 wirelessly or via a wired connection.

[0022] The air pressure sensors 141 are sensors that measure the air pressure of the tires of the drive wheels 120 and the driven wheels 121. For example, an air pressure sensor 141 may be installed on each of the two drive wheels 120 and one driven wheel 121. Each air pressure sensor 141 wirelessly transmits the measured value of the air pressure of the tire to the control unit 130.

[0023] The tire replacement sensor 142 is a sensor that detects that the tire of either the drive wheel 120 or the driven wheel 121 has been replaced. For example, a tire replacement sensor 142 may be installed on each of the two drive wheels 120 and one driven wheel 121. The tire replacement sensor 142 may be an infrared sensor that determines whether a tire is attached to the mobile object 100. Each tire replacement sensor 142 notifies the control unit 130 wirelessly or via a wired connection that a tire has been replaced.

[0024] Next, an example of the functional configuration of the control unit 130 will be described with reference to FIG. 3 . The user instruction acquisition unit 301 acquires user instructions input via the operation unit 133 or the voice input device 135. The user instructions may include a specification of a destination location where the mobile object 100 should arrive. The destination location may be the location of a target object recognized in an image output by a camera and specified by a spoken voice. Furthermore, the destination location may be a geographical location specified by a user using the mobile object 100, or a geographical location that is set in advance by an administrator of the mobile object 100 and stored in the storage device 137. For example, the mobile object 100 may be set to autonomously return to a pre-set geographical location after use by the user.

[0025] The image processing unit 302 recognizes the situation around the mobile object 100 based on the output (e.g., an image) of the camera. For example, the image processing unit 302 recognizes the position, shape, etc. of obstacles around the mobile object 100. The situation around the mobile object 100 may be recognized using point cloud data generated based on the image.

[0026] The map management unit 303 manages a map of the environment in which the mobile object 100 is used. The map may be stored in the storage device 137 of the mobile object 100. The map management unit 303 may receive a map from an external server and store it in the storage device 137.

[0027] The path generating unit 304 generates a path along which the mobile body 100 should travel. For example, when there is no obstacle in the straight direction from the current position to the destination position, the path generating unit 304 generates a path along the straight direction. When there is an obstacle in the straight direction from the current position to the destination position, the path generating unit 304 generates a path so that the mobile body approaches the destination position while avoiding the obstacle. The path generating unit 304 may generate a path so that the angular acceleration of the mobile body 100 is equal to or less than a threshold.

[0028] The travel control unit 305 controls the mobile body 100 to move autonomously along the path generated by the path generation unit 304. If the mobile body 100 is a vehicle, the movement of the mobile body 100 may be expressed as "traveling of the mobile body 100." If the mobile body 100 is a flying object, the movement of the mobile body 100 may be expressed as "flying of the mobile body 100." When the mobile body 100 receives an instruction from a user while traveling, such as to turn right, turn left, or stop, the travel control unit 305 may control the mobile body 100 to travel in accordance with the instruction.

[0029] A method for controlling the traveling of the vehicle 100 will be described with reference to FIG. 4 . Each step of the method in FIG. 4 may be executed by the control unit 130 of the vehicle 100. Specifically, each step of the method in FIG. 4 may be performed by the processor 131 executing a program loaded into the memory device 132. Alternatively, at least some of the steps of the method in FIG. 4 may be executed by a dedicated circuit such as an application-specific integrated circuit (ASIC). The method in FIG. 4 may be initiated in response to an instruction to start traveling control from a passenger or operator of the vehicle 100, or may be initiated after the passenger has finished traveling with the vehicle 100, for example, to automatically return to a standby position. The method in FIG. 4 may be terminated in response to an instruction to end traveling control from a passenger or operator of the vehicle 100, or may be terminated in response to the vehicle 100 reaching a destination position.

[0030] In S401, the control unit 130 (e.g., the image processing unit 302) acquires images captured by a camera. The camera periodically captures images of the surroundings of the moving body 100 (e.g., at 60 frames per second) while the moving body 100 is in operation.

[0031] In S402, the control unit 130 (e.g., the image processing unit 302) generates point cloud data representing the situation around the mobile body 100 based on the image acquired in S401. Point cloud data is data related to a plurality of points located on the surface of one or more objects existing around the mobile body 100. The data related to a point includes the relative position of the point with respect to the mobile body 100. The data related to a point may further include the color of the surface of the object at the point. The control unit 130 generates data related to points on the surface of the object located in each direction for each of a plurality of directions with respect to the mobile body 100. A method for generating point cloud data based on an image may be an existing method, and therefore a detailed description thereof will be omitted.

[0032] In S403, the control unit 130 (e.g., the image processing unit 302) recognizes the situation around the mobile object 100 based on the point cloud data generated in S402. For example, the image processing unit 302 may identify the position and size of an obstacle present around the mobile object 100.

[0033] In S404, the control unit 130 (e.g., the path generation unit 304 and the driving control unit 305) controls the driving of the mobile object 100 based on the surrounding conditions recognized in S403. For example, the path generation unit 304 generates a path for the mobile object 100 so as to avoid obstacles included in the surroundings recognized in S403, and the driving control unit 305 controls the rotational speeds of the two drive wheels 120 of the mobile object 100 so that the mobile object 100 drives along this path. The control unit 130 repeats S401 to S404. For example, the control unit 130 may repeat S401 to S404 at a cycle of 100 ms.

[0034] A method for calibrating a camera of a mobile object 100 will be described with reference to FIG. 5 . Each step of the method of FIG. 5 may be executed by the control unit 130 of the mobile object 100. Specifically, each step of the method of FIG. 5 may be performed by the processor 131 executing a program loaded into the memory device 132. Alternatively, at least some of the steps of the method of FIG. 5 may be executed by a dedicated circuit such as an ASIC. The method of FIG. 5 may be started in response to the power being turned on of the mobile object 100. The method of FIG. 5 may be ended in response to the power being turned off of the mobile object 100. The method of FIG. 5 may be executed during the execution of the method of FIG. 4 or while the method of FIG. 4 is not being executed.

[0035] In S501, the control unit 130 (e.g., the image processing unit 302) determines whether the conditions for calibrating the camera are met. Hereinafter, the conditions for calibrating the camera are referred to as the calibration conditions. If the control unit 130 determines that the calibration conditions are met ("YES" in S501), it transitions the process to S502, and otherwise ("NO" in S501), it repeats S501. In other words, the control unit 130 waits until the calibration conditions are met.

[0036] The calibration condition may include detection of a change in the moving body 100. For example, the control unit 130 may detect that a change has occurred in the moving body 100 based on the amount of impact received by the moving body 100. As described above, the amount of impact received by the moving body 100 is measured by the acceleration sensor 139. The control unit 130 may determine that a change has occurred in the moving body 100 if the amount of impact received by the moving body 100 is greater than a threshold, and may determine that no change has occurred in the moving body 100 if the amount of impact received by the moving body 100 is less than the threshold. Instead of comparing the amount of impact received by the moving body 100 with the threshold, the control unit 130 may compare a value obtained by inputting the amount of impact received by the moving body 100 into a predetermined function with the threshold. If the amount of impact received by the moving body 100 is greater than the threshold, the position of the camera relative to the moving body 100 may change. Therefore, in such a case, the camera can be calibrated at an appropriate timing by calibrating the camera in the following step S505.

[0037] The control unit 130 may detect that a change has occurred in the mobile body 100 based on the amount of luggage carried by the mobile body 100. As described above, the amount of luggage carried by the mobile body 100 is measured by the load sensor 140. The control unit 130 may determine that a change has occurred in the mobile body 100 when the amount of change in the amount of luggage carried by the mobile body 100 is greater than a threshold, and may determine that no change has occurred in the mobile body 100 when the amount of change in the amount of luggage carried by the mobile body 100 is smaller than the threshold. Instead of comparing the amount of change in the amount of luggage carried by the mobile body 100 with the threshold, the control unit 130 may compare a value obtained by inputting the amount of change in the amount of luggage carried by the mobile body 100 into a predetermined function with the threshold. The amount of luggage carried by the mobile body 100 changes when luggage is placed on the loading platform 123 and when luggage is removed from the loading platform 123. When the amount of change in the load of the moving body 100 is greater than a threshold, the position of the camera may change significantly with respect to the ground surface that the tires of the moving body 100 contact. Therefore, in such a case, the camera can be calibrated at an appropriate timing by calibrating the camera in the following step S505.

[0038] The control unit 130 may detect that a change has occurred in the mobile body 100 based on the fact that the tires of the mobile body 100 have been replaced. As described above, the tire replacement sensor 142 detects that the tires of the mobile body 100 have been replaced. The control unit 130 may determine that a change has occurred in the mobile body 100 when the tires of the mobile body 100 have been replaced, and may determine that no change has occurred in the mobile body 100 when the tires of the mobile body 100 have not been replaced. The control unit 130 may determine that a change has occurred in the mobile body 100 when at least one of the multiple tires of the mobile body 100 has been replaced. When the tires of the mobile body 100 have been replaced, the position of the camera relative to the ground that the tires of the mobile body 100 contact may change significantly. Therefore, by calibrating the camera in the following step S505 in such a case, the camera can be calibrated at an appropriate time.

[0039] The control unit 130 may detect that a change has occurred in the mobile object 100 based on the air pressure of the tires of the mobile object 100. As described above, the air pressure of the tires of the mobile object 100 is measured by the air pressure sensor 141. The control unit 130 may determine that a change has occurred in the mobile object 100 when the amount of change in the air pressure of the tires of the mobile object 100 is greater than a threshold, and may determine that no change has occurred in the mobile object 100 when the amount of change in the air pressure of the tires of the mobile object 100 is less than the threshold. The control unit 130 may determine that a change has occurred in the mobile object 100 when the amount of change in the air pressure of at least one of the multiple tires of the mobile object 100 is greater than a threshold. Instead of comparing the amount of change in the air pressure of the tires of the mobile object 100 with the threshold, the control unit 130 may compare a value obtained by inputting the amount of change in the air pressure of the tires of the mobile object 100 into a predetermined function with the threshold. The air pressure of the tires of the moving body 100 changes when air is released from the tires and when air is injected into the tires. If the amount of change in the air pressure of the tires of the moving body 100 is greater than a threshold, the position of the camera relative to the ground with which the tires of the moving body 100 come into contact may change significantly. Therefore, in such a case, by calibrating the camera in the following step S505, the camera can be calibrated at an appropriate timing.

[0040] The control unit 130 may detect a change in the mobile object 100 based on a change in seasons. The change in seasons may cause a change in the temperature around the mobile object 100, which may in turn cause a change in the tire pressure of the mobile object 100. For example, the position of the camera relative to the ground with which the tires of the mobile object 100 contact may differ depending on whether it is summer or winter around the mobile object 100. Therefore, by calibrating the camera in the following step S505 in such a case, the camera can be calibrated at an appropriate time. The control unit 130 may determine the season based on a date. Specifically, the control unit 130 may store a rule defining a period for each season (e.g., June to September for summer) and determine the season based on this rule.

[0041] Based on the above-described multiple conditions, the control unit 130 may detect that a change has occurred in the moving object 100. For example, the control unit 130 may detect that a change has occurred in the moving object 100 when at least one of the above-described multiple conditions is satisfied.

[0042] Steps S502 and S503 may be the same as steps S401 and S402, and therefore redundant description will be omitted. When the method of Fig. 5 is executed in parallel with the method of Fig. 4, steps S502 and S503 may be omitted, and the point cloud data generated in step S402 may be used in subsequent processing.

[0043] In S504, the control unit 130 (for example, the image processing unit 302) detects a plane included in the periphery of the moving object 100 based on the point cloud data generated in S503 (or S402). This plane may be the ground with which the tires of the moving object 100 come into contact.

[0044] In S505, the control unit 130 (e.g., the image processing unit 302) calibrates the camera based on the plane detected in S504. The control unit 130 then transitions the process to S501 and waits until the calibration condition is satisfied again. Calibrating the camera may include changing the internal parameters of the camera or may include changing the parameters used to create point cloud data based on images captured by the camera.

[0045] The plane detection in S504 and the camera calibration in S505 will be described with reference to FIG. 6 . Point cloud data 600 represents the point cloud data generated in S503. The point cloud data 600 includes the relative positions of each of a plurality of points 601 with respect to the moving body 100. In FIG. 6 , only one of the plurality of points 601 is denoted by a reference symbol. The plurality of points 601 are arranged in a three-dimensional Cartesian coordinate system CS (hereinafter simply referred to as the coordinate system CS) based on the position of the moving body 100. The x-axis of the coordinate system CS indicates the front-to-rear direction of the moving body 100, with the forward direction of the moving body 100 being positive. The y-axis of the coordinate system CS indicates the width direction of the moving body 100. The z-axis of the coordinate system CS indicates the up-down direction of the moving body 100, with the upward direction of the moving body 100 being positive. The origin of the coordinate system CS represents the position of the moving body 100.

[0046] The control unit 130 generates point cloud data 610 by performing equal-interval downsampling and filtering on the point cloud data 600. Since equal-interval downsampling and filtering may be performed using existing methods, detailed description thereof will be omitted. The number of points included in the point cloud data 610 is smaller than the number of points included in the point cloud data 600.

[0047] Next, the control unit 130 generates point cloud data 620 by extracting data relating to points included in a specific region 602 from the point cloud data 610. The region 602 is an area where the ground is likely to exist in front of the mobile object 100. For example, the region 602 may be an area where the x component falls within a range of 0 m to 2 m, the y component falls within a range of −1 m to 1 m, and the z component falls within a range of −1 m to 1 m.

[0048] Next, the control unit 130 calculates an equation of a plane that fits multiple points included in the point cloud data 620. This fitting may be performed using an existing method such as the least-squares method. Then, the control unit 130 determines the offset amount in the z-axis direction, the roll angle (i.e., the amount of rotation around the x-axis), and the pitch angle (i.e., the amount of rotation around the y-axis) so that this plane passes through the origin of the coordinate system CS and the normal vector of this plane is parallel to the z-axis. The control unit 130 calibrates the camera based on these values.

[0049] If a plane fitting a plurality of points included in the point cloud data 620 satisfies a condition for calibrating the camera, the control unit 130 may calibrate the camera based on this plane. Hereinafter, the condition for calibrating the camera is referred to as a planarity condition. If the plane does not satisfy the planarity condition, the control unit 130 may not calibrate the camera. The planarity condition may be that an object included in the region 602 is an approximately horizontal plane. For example, the control unit 130 may determine that an object included in the region 602 is not an approximately horizontal plane if the error between a plurality of points included in the point cloud data 620 and the plane fitting these points is greater than a threshold. The control unit 130 may determine that an object included in the region 602 is not an approximately horizontal plane if the angle between the up-down direction of the mobile object 100 and the direction of gravity is greater than a threshold. The gravity direction is measured, for example, by the acceleration sensor 139. In this way, by performing calibration when it is determined that the mobile object 100 is located on an approximately horizontal plane, the calibration can be performed appropriately.

[0050] An example of point cloud data before and after calibration will be described with reference to Figure 7. Image 701 is a diagram of the point cloud data before calibration projected onto the yz plane, and image 702 is a diagram of the point cloud data before calibration projected onto the xz plane. Image 711 is a diagram of the point cloud data after calibration projected onto the yz plane, and image 712 is a diagram of the point cloud data after calibration projected onto the xz plane. As can be seen from Figure 7, the horizontal plane (ground) is accurately recognized in the point cloud data after calibration.

[0051] In the above example, the method of Fig. 5 is executed by the control unit 130 of the mobile object 100. Alternatively, the method of Fig. 5 may be executed by a device external to the mobile object 100 (e.g., a server communicating with the mobile object 100 via a network). In this case, calibrating the camera in S505 of Fig. 5 may involve the external device transmitting parameters for calibrating the camera of the mobile object 100 to the mobile object 100 and causing the mobile object 100 to calibrate the camera.

[0052] Summary of the embodiment (Item 1) A calibration device (130) comprising: an acquisition unit that acquires images of the periphery of a moving object (100) captured by cameras (114-116) of the moving object; a point cloud generation unit that generates point cloud data (600) representing the situation around the moving object based on the images; a plane detection unit that detects planes included in the periphery of the moving object based on the point cloud data; a change detection unit that detects a change in the moving object; and a calibration unit that, when a change in the moving object is detected, calibrates the camera based on the planes. According to this item, the camera can be calibrated at an appropriate time. (Item 2) The moving object comprises a sensor (139) that measures an impact received by the moving object, and the change detection unit detects a change in the moving object based on the amount of impact received by the moving object. The calibration device according to item 1, wherein the moving object comprises a sensor (139) that measures an impact received by the moving object, and the change detection unit detects a change in the moving object based on the amount of impact received by the moving object. According to this item, the camera can be calibrated when the position of the camera changes. (Item 3) The calibration device according to item 1 or 2, wherein the mobile body includes a sensor (140) that measures a load of luggage on the mobile body, and the change detection unit detects that a change has occurred in the mobile body based on the load of luggage on the mobile body. According to this item, the camera can be calibrated when the load of luggage changes. (Item 4) The calibration device according to any one of items 1 to 3, wherein the mobile body includes a sensor (142) that detects that tires of the mobile body have been replaced, and the change detection unit detects that a change has occurred in the mobile body based on the tire replacement. According to this item, the camera can be calibrated when the tires are replaced. (Item 5) The calibration device according to any one of items 1 to 4, wherein the mobile body includes a sensor (141) that measures air pressure in the tires of the mobile body, and the change detection unit detects that a change has occurred in the mobile body based on the air pressure in the tires of the mobile body. According to this item, the camera can be calibrated when the air pressure changes. (Item 6) The calibration device according to any one of Items 1 to 5, wherein the change detection unit detects that a change has occurred in the moving object based on a change in season.According to this item, the camera can be calibrated at an appropriate time based on seasonal changes. (Item 7) The calibration device according to any one of items 1 to 6, wherein the calibration unit calibrates the camera based on the plane if the plane satisfies a condition for calibrating the camera. According to this item, the camera can be calibrated appropriately. (Item 8) A mobile body (100) including the calibration device (130) according to any one of items 1 to 7. According to this item, the camera can be calibrated at an appropriate time. (Item 9) A program for causing a computer (130) to execute the following operations: acquiring images of the periphery of a moving object (100) captured by cameras (114-116) of the moving object (S502), generating point cloud data (600) representing the situation around the moving object based on the images (S503), detecting planes included in the periphery of the moving object based on the point cloud data (S504), detecting that a change has occurred in the moving object (S501), and calibrating the camera based on the planes when a change has occurred in the moving object (S505). According to this item, the camera can be calibrated at an appropriate timing. (Item 10) A method for controlling a calibration device (130), comprising: acquiring an image of a periphery of a moving object (100) captured by a camera (114-116) of the moving object (S502); generating point cloud data (600) representing a situation around the moving object based on the image (S503); detecting a plane included in the periphery of the moving object based on the point cloud data (S504); detecting that a change has occurred in the moving object (S501); and calibrating the camera based on the plane when a change has occurred in the moving object (S505). According to this item, the camera can be calibrated at an appropriate time.

[0053] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. A calibration device comprising: an acquisition unit that acquires images of the surroundings of a moving body captured by a camera of the moving body; a point cloud generation unit that generates point cloud data representing the situation around the moving body based on the images; a plane detection unit that detects planes included in the surroundings of the moving body based on the point cloud data; a change detection unit that detects that a change has occurred in the moving body; and a calibration unit that calibrates the camera based on the planes when a change has been detected in the moving body.

2. The calibration device according to claim 1, wherein the moving body is equipped with a sensor that measures an impact received by the moving body, and the change detection unit detects that a change has occurred in the moving body based on the amount of impact received by the moving body.

3. A calibration device as described in claim 1 or 2, wherein the moving body is equipped with a sensor that measures the amount of luggage carried on the moving body, and the change detection unit detects that a change has occurred in the moving body based on the amount of luggage carried on the moving body.

4. A calibration device as claimed in any one of claims 1 to 3, wherein the mobile body is equipped with a sensor that detects that the tires of the mobile body have been replaced, and the change detection unit detects that a change has occurred in the mobile body based on the tires of the mobile body being replaced.

5. A calibration device according to any one of claims 1 to 4, wherein the moving body is equipped with a sensor that measures the air pressure of the tires of the moving body, and the change detection unit detects that a change has occurred in the moving body based on the air pressure of the tires of the moving body.

6. A calibration device according to any one of claims 1 to 5, wherein the change detection unit detects that a change has occurred in the moving object based on a change in season.

7. The calibration device according to any one of claims 1 to 6, wherein the calibration unit calibrates the camera based on the plane if the plane satisfies a condition for calibrating the camera.

8. A mobile object equipped with a calibration device according to any one of claims 1 to 7.

9. A program for causing a computer to perform the following operations: acquiring images of the surroundings of a moving body taken by a camera of the moving body; generating point cloud data representing the situation around the moving body based on the images; detecting planes contained in the surroundings of the moving body based on the point cloud data; detecting that a change has occurred in the moving body; and calibrating the camera based on the planes when a change has occurred in the moving body is detected.

10. A method for controlling a calibration device, comprising: acquiring an image of the surroundings of a moving body taken by a camera of the moving body; generating point cloud data representing the situation around the moving body based on the image; detecting a plane contained in the surroundings of the moving body based on the point cloud data; detecting that a change has occurred in the moving body; and calibrating the camera based on the plane when a change has occurred in the moving body is detected.

Images