Imaging control device and imaging control method

Abstract

An exposure control device of the present disclosure comprises: a setting unit that calculates camera angles between a moving direction of an own device and an optical axis direction of each of a plurality of stereo cameras, and sets exposure frequencies of the plurality of respective stereo cameras on the basis of the camera angles; and an exposure control unit that controls the operations of the plurality of stereo cameras on the basis of the exposure frequencies set by the setting unit.

Description

Technical Field

[0001] The present disclosure relates to an imaging control device that controls the operation of an imaging device, and an imaging control method used in such an imaging control device.

Background Art

[0002] In a robot device, an imaging device is often provided, and map data indicating the environment around the robot device is generated based on an image obtained by the imaging device. For example, Patent Document 1 discloses a robot device that generates an environmental map and determines an action using the generated environmental map.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] By the way, when performing arithmetic processing based on an image obtained by an imaging device, it is desirable to be able to effectively use arithmetic resources.

[0005] It is desirable to provide an imaging control device and an imaging control method that can effectively use arithmetic resources.

[0006] The imaging control device according to an embodiment of the present disclosure includes a setting unit and an exposure control unit. The setting unit Provided on the moving body calculates a camera angle between the moving direction of its own device and the optical axis direction of each of a plurality of stereo cameras, and is configured to set the exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles. The exposure control unit is configured to control the operation of the plurality of stereo cameras based on the plurality of exposure frequencies set by the setting unit. Provided on the moving body such that the optical axis directions are different from each other

[0007] ​The imaging control method according to an embodiment of the present disclosure includes: Provided on the moving body calculating the camera angles between the moving direction of the own device and the optical axis directions of each of the plurality of stereo cameras, and setting the exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles; and Provided on the moving body such that the optical axis directions are different from each other controlling the operations of the plurality of stereo cameras based on the set plurality of exposure frequencies.

[0008] In the imaging control device and the imaging control method according to an embodiment of the present disclosure, the camera angles between the moving direction of the own device and the optical axis directions of each of the plurality of stereo cameras are calculated, and the exposure frequency of each of the plurality of stereo cameras is set based on the plurality of camera angles. Then, the operations of the plurality of stereo cameras are controlled based on the set plurality of exposure frequencies.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order. 1. First Embodiment 2. Second Embodiment 3. Third Embodiment 4. Fourth Embodiment

[0011] <1. First Embodiment> [Configuration Example] FIG. 1 shows a configuration example of a drone 1 equipped with an imaging control device according to an embodiment. FIG. 2 shows an example of the appearance of the drone 1, where FIG. 2(A) shows a plan view and FIG. 2(B) shows a front view. Note that the imaging control method according to the embodiment of the present disclosure is implemented by this embodiment and will be described together.

[0012] As shown in FIG. 1, the drone 1 includes an imaging unit 11, a GPS (Global Positioning System) reception unit 12, a barometric pressure sensor 13, an inertial measurement unit 14, and a control unit 20. Also, as shown in FIG. 2, the drone 1 includes a main body unit 91, four arm units 92, and four rotary wings 93. The imaging unit 11, the GPS reception unit 12, the barometric pressure sensor 13, the inertial measurement unit 14, and the control unit 20 are housed in the main body unit 91 of the drone 1.

[0013] The imaging unit 11 (FIG. 1) is configured to generate image data by imaging the surroundings of the drone 1 based on an instruction from the control unit 20. As shown in FIG. 2, in this example, the imaging unit 11 has six stereo cameras 100 (stereo cameras 100A to 100F). Each of the six stereo cameras 100 has, in this example, two image sensors arranged at a predetermined distance from each other, and is configured to generate image data (stereo image data) for two captured images with a parallax by performing an exposure operation. This stereo image data has a time stamp indicating the exposure timing. Note that, in this example, each of the six stereo cameras 100 has two image sensors, but it is not limited to this, and for example, it may have three or more image sensors.

[0014] As shown in FIG. 2, the six stereo cameras 100 are arranged in the main body 91 such that the optical axis directions 101 are different from each other. Specifically, the optical axis direction 101A of the stereo camera 100A is the X-axis direction, the optical axis direction 101B of the stereo camera 100B is the Y-axis direction, the optical axis direction 101C of the stereo camera 100C is the direction opposite to the X-axis direction, the optical axis direction 101D of the stereo camera 100D is the direction opposite to the Y-axis direction, the optical axis direction 101E of the stereo camera 100E is the Z-axis direction, and the optical axis direction 101F of the stereo camera 100F is the direction opposite to the Z-axis direction. With this configuration, the stereo cameras 100A to 100D image the horizontal direction of the drone 1, the stereo camera 100E images the upward direction of the drone 1, and the stereo camera 100F images the downward direction of the drone 1.

[0015] FIG. 3 shows a configuration example of the imaging unit 11. The imaging unit 11 includes six stereo cameras 100 and a bus wiring 109. The six stereo cameras 100 are connected to the bus wiring 109. The bus wiring 109 is configured to include, for example, a plurality of wirings. The communication interface of the six stereo cameras 100 can apply, for example, MIPI (Mobile Industry Processor Interface). The six stereo cameras 100 transmit the generated image data to the control unit 20 in a time-division manner via the bus wiring 109.

[0016] The GPS receiver 12 is configured to detect the position of the drone 1 by receiving signals transmitted from a plurality of artificial satellites. The pressure sensor 13 is configured to detect the atmospheric pressure. Information about the detected atmospheric pressure is used to detect the altitude of the drone 1. In this way, the GPS receiver 12 and the pressure sensor 13 function as position sensors for detecting the position of the drone 1.

[0017] The inertial measurement unit 14 is configured using, for example, an IMU (inertial measurement unit) and is configured to detect angular velocity and acceleration. The IMU includes, for example, an acceleration sensor, a gyro sensor, a magnetic sensor, and the like.

[0018] The control unit 20 is configured to control the flight of the drone 1 based on the data supplied from the imaging unit 11, the GPS reception unit 12, the pressure sensor 13, and the inertial measurement unit 14. In addition, the control unit 20 also performs processing to control the operation of the imaging unit 11. The control unit 20 includes an image data acquisition unit 21, a position data acquisition unit 22, an inertial data acquisition unit 23, a depth data generation unit 24, a position movement estimation unit 25, an exposure frequency setting unit 26, a timing control unit 27, a map data generation unit 31, a behavior planning unit 32, a route determination unit 33, and an aircraft control unit 34. The control unit 20 is configured using, for example, one or more processors.

[0019] The image data acquisition unit 21 acquires the image data supplied from the imaging unit 11 and supplies the acquired data as image data DT to the depth data generation unit 24 and the position movement estimation unit 25.

[0020] The position data acquisition unit 22 acquires the detection results of the GPS reception unit 12 and the pressure sensor 13 and supplies the acquired data as position data DP to the position movement estimation unit 25.

[0021] The inertial data acquisition unit 23 acquires the detection results of the inertial measurement unit 14 and supplies the acquired data as inertial data DI to the position movement estimation unit 25.

[0022] The depth data generation unit 24 is configured to generate depth data DD including a depth map indicating a map of depth values in the captured image based on the stereo image data included in the image data DT.

[0023] The position and motion estimation unit 25 is configured to estimate the position and motion of the drone 1 based on the image data DT, the position data DP, and the inertial data DI. Specifically, the position and motion estimation unit 25 performs VIO (Visual Inertial Odometry) processing based on the image data DT, and uses the position data DP and the inertial data DI to calculate the position, attitude, velocity vector VS, and angular velocity of the drone 1 at the time indicated by the timestamp included in the image data DT, and generates position and motion data DPM including these data.

[0024] The exposure frequency setting unit 26 is configured to set the exposure frequencies of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position and motion data DPM, and to set the respective exposure timings of the six stereo cameras 100 based on the set exposure frequencies. Specifically, the exposure frequency setting unit 26 sets the exposure frequencies of the six stereo cameras 100 based on the velocity value indicated by the velocity vector VS and the angle θ between the flight direction DIR of the drone 1 indicated by the velocity vector VS and the optical axis direction 101 of the six stereo cameras 100. Then, the exposure frequency setting unit 26 is configured to set the respective exposure timings of the six stereo cameras 100 so that the exposure timings of the six stereo cameras 100 are different from each other.

[0025] The timing control unit 27 is configured to control the operation timings of the six stereo cameras 100 based on an instruction from the exposure frequency setting unit 26.

[0026] The map data generation unit 31 is configured to generate map data MAP including an occupancy map based on the depth data DD and the position and motion data DPM. Specifically, the map data generation unit 31 generates the map data MAP by performing a voting process on each voxel in the occupancy map based on the depth value included in the depth data DD and the information about the position of the drone 1 included in the position and motion data DPM.

[0027] The action planning unit 32 is configured to plan the actions of the drone 1 based on a preset destination or on operation instruction data transmitted from the operation terminal.

[0028] The route determination unit 33 is configured to determine the flight route of the drone 1 based on the map data MAP and the determination result of the action planning unit 32.

[0029] The airframe control unit 34 is configured to control the airframe of the drone 1 based on the flight route determined by the route determination unit 33. As shown in FIG. 2, the four rotors 93 are respectively attached to the tips of the four arm portions 92 and are configured to be rotatable. The four rotors 93 rotate based on the driving force generated by four motors (not shown). The airframe control unit 34 controls the operations of these four motors based on the determined flight route. Thereby, the drone 1 can fly along the determined flight route.

[0030] Here, the exposure frequency setting unit 26 corresponds to a specific example of the "setting unit" in the present disclosure. The timing control unit 27 corresponds to a specific example of the "exposure control unit" in the present disclosure. The six stereo cameras 100 correspond to a specific example of the "plurality of stereo cameras" in the present disclosure. The angle θ corresponds to a specific example of the "camera angle" in the present disclosure. The bus wiring 109 corresponds to a specific example of the "bus wiring" in the present disclosure. The image data acquisition unit 21 corresponds to a specific example of the "image acquisition unit" in the present disclosure.

[0031] [Operations and Effects] Subsequently, the operations and effects of the drone 1 of the present embodiment will be described.

[0032] (Overall Operation Outline) First, referring to FIG. 1, the overall operation overview of the drone 1 will be described. The imaging unit 11 generates image data by imaging the surroundings of the drone 1 based on an instruction from the control unit 20. The GPS receiving unit 12 detects the position of the drone 1 by receiving signals transmitted from a plurality of artificial satellites. The air pressure sensor 13 detects the air pressure. The inertial measurement unit 14 detects the angular velocity and acceleration. The control unit 20 controls the flight of the drone 1 based on the data supplied from the imaging unit 11, the GPS receiving unit 12, the air pressure sensor 13, and the inertial measurement unit 14, and also controls the operation of the imaging unit 11.

[0033] (Detailed operation) FIG. 4 shows an operation example of the control unit 20. In the control unit 20, the depth data generation unit 24 generates depth data DD based on the stereo image data included in the image data DT, and the position movement estimation unit 25 generates position movement data DPM by estimating the position and movement of the drone 1 based on the image data DT, the position data DP, and the inertial data DI. The exposure frequency setting unit 26 sets the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM, and sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency. The map data generation unit 31 generates map data MAP including an occupancy map based on the depth data DD and the position movement data DPM. The following will explain this operation in detail.

[0034] The position movement estimation unit 25 calculates the velocity vector VS of the drone 1 based on the image data DT, the position data DP, and the inertial data DI (step S101). Specifically, the position movement estimation unit 25 performs VIO processing based on the image data DT and uses the position data DP and the inertial data DI to calculate the velocity vector VS of the drone 1.

[0035] Next, the exposure frequency setting unit 26 checks whether the velocity value indicated by the velocity vector VS is less than or equal to the threshold value TH1 (step S102).

[0036] In step S102, when the speed value is not less than the threshold value TH1 (\"N\" in step S102), the exposure frequency setting unit 26 calculates the angle θ between the flight direction DIR of the drone 1 indicated by the speed vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103).

[0037] FIG. 5 shows an example of the angle θ. In this example, the flight direction DIR is the horizontal direction, and it is the direction closer to the optical axis direction 101A between the optical axis direction 101A of the stereo camera 100A and the optical axis direction 101B of the stereo camera 100B. The exposure frequency setting unit 26 calculates the angle θA between the flight direction DIR and the optical axis direction 101A of the stereo camera 100A, calculates the angle θB between the flight direction DIR and the optical axis direction 101B of the stereo camera 100B, calculates the angle θC between the flight direction DIR and the optical axis direction 101C of the stereo camera 100C, and calculates the angle θD between the flight direction DIR and the optical axis direction 101D of the stereo camera 100D. Although not shown, the exposure frequency setting unit 26 calculates the angle θE between the flight direction DIR and the optical axis direction 101E of the stereo camera 100E, and calculates the angle θF between the flight direction DIR and the optical axis direction 101F of the stereo camera 100F. The angle θ is an angle smaller than 180 degrees.

[0038] In this example, since the flight direction DIR is a direction close to the optical axis direction 101A of the stereo camera 100A, the angle θA is the smallest and the angle θC is the largest. Since the flight direction DIR is the horizontal direction, the angles θE and θF are 90 degrees.

[0039] Next, the exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100 based on the six angles θ (step S104). Specifically, the exposure frequency setting unit 26 sets the exposure frequency of the stereo camera 100 corresponding to the smallest angle θ among the six angles θ to be higher than the exposure frequency of any one of the other stereo cameras 100. That is, the exposure frequency setting unit 26 sets the exposure frequency so as to increase the exposure frequency of the stereo camera 100 that images the flight direction DIR in order to more accurately grasp the environment in the flight direction DIR.

[0040] For example, the exposure frequency setting unit 26 can calculate the cosine values of the six angles θ respectively, and set the exposure frequency of each of the six stereo cameras 100 based on the cosine values. For example, when the flight direction DIR is the same as the optical axis direction 101A of the stereo camera 100A, the angles θA to θF are as follows. Stereo camera 100A: θA = 0 degrees Stereo camera 100B: θB = 90 degrees Stereo camera 100C: θC = 180 degrees Stereo camera 100D: θD = 90 degrees Stereo camera 100E: θE = 90 degrees Stereo camera 100F: θF = 90 degrees Therefore, the cosine values are as follows. Stereo camera 100A: cosθA = 1 Stereo camera 100B: cosθB = 0 Stereo camera 100C: cosθC = -1 Stereo camera 100D: cosθD = 0 Stereo camera 100E: cosθE = 0 Stereo camera 100F: cosθF = 0 In this example, since cosθC is the smallest, these cosine values are divided by the absolute value of this cosθC, and 2 is added to the result. Thereby, the following numerical values are obtained Stereo camera 100A: 3 Stereo camera 100B: 2 Stereo camera 100C: 1 Stereo camera 100D: 2 Stereo camera 100E: 2 Stereo camera 100F: 2 In this example, by squaring these values, the exposure frequency of each of the six stereo cameras 100 can be set as follows. Stereo camera 100A: 9 Stereo camera 100B: 4 Stereo camera 100C: 1 Stereo camera 100D: 4 Stereo camera 100E: 4 Stereo camera 100F: 4 That is, in this example, the ratio of the exposure frequency of stereo camera 100A, the exposure frequencies of stereo cameras 100B, 100D, 100E, and 100F, and the exposure frequency of stereo camera 100C can be made 9:4:1. Note that in this example, the values are squared, but if it is desired to reduce the difference in exposure frequencies, it is not necessary to square the values. In this case, the ratio of the exposure frequencies will be 3:2:1. Also, if it is desired to increase the difference in exposure frequencies, the values may be cubed. In this case, the ratio of the exposure frequencies will be 27:8:1.

[0041] Note that the method of setting the exposure frequency of each of the six stereo cameras 100 based on the six angles θ described above is an example and is not limited thereto, and various methods can be used.

[0042] In step S102, when the speed value is less than or equal to the threshold value TH1 (in step S102, “Y”), the exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100 to the same frequency (step S105). That is, the exposure frequency setting unit 26 sets the exposure frequencies of all the stereo cameras 100 to the same frequency in order to grasp the overall environment around the drone 1.

[0043] In this way, the exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100. Then, based on the set exposure frequency, the exposure frequency setting unit 26 sets the exposure timing of each of the six stereo cameras 100. At this time, the exposure frequency setting unit 26 sets the exposure timing of each of the six stereo cameras 100 so that the exposure timings in the six stereo cameras 100 are different from each other.

[0044] Next, the timing control unit 27 controls the operation timing of the six stereo cameras 100 based on an instruction from the exposure frequency setting unit 26 (step S106).

[0045] Also, the depth data generation unit 24 generates depth data DD based on the stereo image data included in the image data DT (step S107).

[0046] Next, the position movement estimation unit 25 calculates the position of the drone 1 at the time indicated by the timestamp based on the timestamp included in the image data DT (step S108).

[0047] Then, the map data generation unit 31 performs a voting process on each voxel in the occupancy map based on the position of the drone 1 obtained in step S108 and the depth value obtained from the stereo image data including the timestamp. In this way, the map data MAP is updated.

[0048] Thus, this flow ends. The control unit 20 repeats such processing for each frame period.

[0049] In this way, in the drone 1, based on six angles θ, the exposure frequency of each of the six stereo cameras 100 is set. As a result, in the drone 1, the computing resources in the control unit 20 can be effectively utilized. That is, in order to more accurately grasp the environment around the drone 1, it is desirable to have a high exposure frequency. However, in this case, the computational load on the control unit 20 will increase. In the drone 1, based on six angles θ, the exposure frequency of each of the six stereo cameras 100 is set. Thereby, for example, the exposure frequency of the stereo camera 100 that images the flight direction DIR can be increased, and the exposure frequency of the stereo camera 100 that images the direction opposite to the flight direction DIR can be decreased. As a result, in the drone 1, since a large amount of the computing resources of the control unit 20 can be allocated to the computational processing of the image data obtained by imaging the flight direction DIR, while effectively utilizing the limited computing resources, the environment in the flight direction DIR can be grasped more accurately.

[0050] Also, in the drone 1, when the flight speed is faster than a predetermined speed, based on six angles θ, the exposure frequency of each of the six stereo cameras 100 is set. As a result, in the drone 1, the computing resources in the control unit 20 can be effectively utilized. That is, when the flight speed is slow, since the movement amount of the drone 1 is small, even if the exposure frequency is low, the environment around the drone 1 can be grasped accurately to a certain extent. On the other hand, when the flight speed is fast, since the movement amount of the drone 1 is large, in order to accurately grasp the environment around the drone 1, it is desirable to have a high exposure frequency. However, in this case, the computational load on the control unit 20 will increase. In the drone 1, when the flight speed is fast, based on six angles θ, the exposure frequency of each of the six stereo cameras 100 is set. Thereby, in the drone 1, when the flight speed is fast, since a large amount of the computing resources of the control unit 20 can be allocated to the computational processing of the image data obtained by imaging the flight direction DIR, the limited computing resources can be effectively utilized.

[0051] (Regarding Exposure Timing) The exposure frequency setting unit 26 sets the exposure timing of each of the six stereo cameras 100 so that the exposure timings in the six stereo cameras 100 are different from each other. The six stereo cameras 100 generate image data by performing an exposure operation based on an instruction from the control unit 20. Then, the six stereo cameras 100 transmit the generated image data to the control unit 20 in a time-division manner via the bus wiring 109. The following will explain this operation in detail.

[0052] FIG. 6 shows an operation example of the imaging unit 11. In this example, the exposure timing is set in the order of stereo camera 100A, stereo camera 100B, stereo camera 100C,....

[0053] First, the two image sensors 100A1 and 100A2 of the stereo camera 100A perform an exposure operation EX in the period from timing t11 to t12 based on the synchronization signal SYNC1. Then, the stereo camera 100A performs a transmission operation TX in the period from timing t12 to t13 to transmit the image data generated by the image sensor 100A1 to the control unit 20 via the bus wiring 109, and performs a transmission operation TX in the period from timing t13 to t14 to transmit the image data generated by the image sensor 100A2 to the control unit 20 via the bus wiring 109.

[0054] Similarly, the two image sensors 100B1 and 100B2 of the stereo camera 100B perform an exposure operation EX in the period from timing t21 to t22 based on the synchronization signal SYNC2. Then, the stereo camera 100B performs a transmission operation TX in the period from timing t22 to t23 to transmit the image data generated by the image sensor 100B1 to the control unit 20 via the bus wiring 109, and performs a transmission operation TX in the period from timing t23 to t24 to transmit the image data generated by the image sensor 100B2 to the control unit 20 via the bus wiring 109.

[0055] Similarly, the two image sensors 100C1 and 100C2 of the stereo camera 100C perform an exposure operation EX during periods from timing t31 to t32 based on the synchronization signal SYNC3. Then, the stereo camera 100C performs a transmission operation TX during the period from timing t32 to t33, and transmits the image data generated by the image sensor 100C1 to the control unit 20 via the bus wiring 109. The stereo camera 100C performs a transmission operation TX during the period from timing t33 to t34, and transmits the image data generated by the image sensor 100C2 to the control unit 20 via the bus wiring 109.

[0056] In this way, for the six stereo cameras 100, the exposure timings are set to be different from each other. Then, the six stereo cameras 100 transmit the generated image data to the control unit 20 in a time-division manner via the bus wiring 109.

[0057] In this way, in the drone 1, since the exposure timings of the six stereo cameras 100 are different from each other, delay can be suppressed. That is, for example, when the exposure timings of the six stereo cameras 100 are the same, as shown in FIG. 7, for example, in the stereo cameras 100B and 100C, a time td from the exposure operation EX to the transmission operation TX occurs, and there is a possibility that the control unit 20 cannot acquire the image data in a timely manner. On the other hand, in the drone 1, since the exposure timings of the six stereo cameras 100 are different from each other, for example, as shown in FIG. 6, the transmission operation TX can be performed in a short time after the exposure operation EX, so that the control unit 20 can acquire the image data in a timely manner. Thereby, in the drone 1, for example, the accuracy of arithmetic processing can be improved.

[0058] (Regarding the exposure frequency) The exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100 based on the six angles θ. The six stereo cameras 100 generate image data by performing an exposure operation based on an instruction from the control unit 20. The control unit 20 performs processing based on the image data generated by the six stereo cameras 100.

[0059] FIG. 8 shows an operation example of the control unit 20. (A) to (F) show the operations of the stereo cameras 100A, 100B, 100D, 100E, 100F, and 100C, respectively. (G) shows the operation of the position movement estimation unit 25, and (H) shows the operations of the depth data generation unit 24 and the map data generation unit 31. In this example, the ratio of the exposure frequency of the stereo camera 100A, the exposure frequencies of the stereo cameras 100B, 100D, 100E, and 100F, and the exposure frequency of the stereo camera 100C is 4:2:1.

[0060] In this example, as shown in FIGS. 8(A) to (F), the imaging unit 11 performs the exposure operation in the order of the stereo camera 100A, the stereo camera 100B, the stereo camera 100D, the stereo camera 100A, the stereo camera 100E, the stereo camera 100F,.... As a result, the stereo camera 100A performs the exposure operation at a rate of 8 times in 26 frames, the stereo cameras 100B, 100D, 100E, and 100F perform the exposure operation at a rate of 4 times in 26 frames, and the stereo camera 100C performs the exposure operation at a rate of 2 times in 26 frames.

[0061] The position movement estimation unit 25 performs processing based on the supplied image data DT. That is, as shown in FIG. 8(G), the position movement estimation unit 25 first performs processing based on the image data generated by the stereo camera 100A, then performs processing based on the image data generated by the stereo camera 100B, performs processing based on the image data generated by the stereo camera 100D, performs processing based on the image data generated by the stereo camera 100A, performs processing based on the image data generated by the stereo camera 100E, and performs processing based on the image data generated by the stereo camera 100F. The same applies thereafter.

[0062] The depth data generation unit 24 and the map data generation unit 31 perform processing based on the supplied image data DT. That is, as shown in FIG. 8(H), the depth data generation unit 24 and the map data generation unit 31 first perform processing based on the image data generated by the stereo camera 100A, then perform processing based on the image data generated by the stereo camera 100B, perform processing based on the image data generated by the stereo camera 100D, perform processing based on the image data generated by the stereo camera 100A, perform processing based on the image data generated by the stereo camera 100E, and perform processing based on the image data generated by the stereo camera 100F. The same applies thereafter.

[0063] In this way, in the drone 1, each of the six stereo cameras 100 performs an exposure operation at substantially equal time intervals. Specifically, for example, the stereo camera 100A performs an exposure operation at a rate of once every 3 to 4 frames, the stereo cameras 100B, 100D, 100E, and 100F perform an exposure operation at a rate of once every 6 to 7 frames, and the stereo camera 100C performs an exposure operation at a rate of once every 13 frames. Thereby, the drone 1 can accurately grasp the surrounding environment.

[0064] [Effect] As described above, in the present embodiment, since the exposure frequency of each of the six stereo cameras is set based on the six angles θ, the calculation resources can be effectively utilized.

[0065] In the present embodiment, when the flight speed is higher than a predetermined speed, since the exposure frequency of each of the six stereo cameras is set based on the six angles θ, the calculation resources can be effectively utilized.

[0066] In the present embodiment, since the exposure timings of the six stereo cameras are different from each other, the delay can be suppressed, and for example, the accuracy of arithmetic processing can be improved.

[0067] [Modification Example 1-1] In the above embodiment, the exposure frequency setting unit 26 increased the exposure frequency of the stereo camera 100 corresponding to the smallest angle θ among the six angles θ. At that time, the exposure frequency setting unit 26 may increase the exposure frequency as the speed value indicated by the velocity vector VS is larger. Specifically, for example, when the flight direction DIR is the same as the optical axis direction 101A of the stereo camera 100A, when the speed is slow, the ratio of the exposure frequency of the stereo camera 100A, the exposure frequencies of the stereo cameras 100B, 100D, 100E, 100F, and the exposure frequency of the stereo camera 100C is 3:2:1, and when the speed is fast, the ratio of the exposure frequency of the stereo camera 100A, the exposure frequencies of the stereo cameras 100B, 100D, 100E, 100F, and the exposure frequency of the stereo camera 100C can be 9:4:1.

[0068] [Modification Example 1-2] In the above-described embodiment, the depth data generation unit 24 and the map data generation unit 31 perform processing in each frame, but the present invention is not limited thereto. Instead, for example, as shown in FIG. 9, there may be a frame in which the depth data generation unit 24 and the map data generation unit 31 do not perform processing. In this example, the position movement estimation unit 25 performs processing in each frame. Further, the depth data generation unit 24 and the map data generation unit 31 do not perform processing at a rate of once every 6 to 7 frames. Further, for example, there may be a frame in which the position movement estimation unit 25 does not perform processing. In this example, the depth data generation unit 24 and the map data generation unit 31 perform processing in each frame.

[0069] [Modification Example 1-3] In the above embodiment, for example, as shown in FIG. 8, the position movement estimation unit 25, the depth data generation unit 24, and the map data generation unit 31 perform processing based on the same image data. However, the present invention is not limited to this. Instead, for example, as shown in FIG. 10, the image data on which the position movement estimation unit 25 performs processing and the image data on which the depth data generation unit 24 and the map data generation unit 31 perform processing may be different from each other. FIGS. 10(A) to (F) respectively show the operations of the stereo cameras 100A, 100B, 100D, 100E, 100F, and 100C. FIG. 10(G) shows the operation of the position movement estimation unit 25, and FIG. 10(H) shows the operations of the depth data generation unit 24 and the map data generation unit 31. In FIGS. 10(A) to (F), the solid line indicates the stereo camera 100 that generates the image data on which the depth data generation unit 24 and the map data generation unit 31 perform processing, and the dashed line indicates the stereo camera 100 that generates the image data on which the position movement estimation unit 25 performs processing. In this example, the position movement estimation unit 25 performs VIO processing at a rate of about once every two frames based on the image data generated by the stereo camera 100F that images the downward direction of the drone 1. For example, when the exposure timings of the two stereo cameras 100 coincide, as shown in FIG. 7, the timing of the transmission operation TX is shifted. For example, when the size of the image is small or there is a margin in the communication data amount in the bus wiring 109, the exposure timings of the plurality of stereo cameras 100 may coincide in this way.

[0070] [Modification Example 1-4] In the above embodiment, the control unit 20 of the drone 1 controls the flight of the drone 1 based on the data supplied from the imaging unit 11, the GPS reception unit 12, the pressure sensor 13, and the inertial measurement unit 14, and performs the process of controlling the operation of the imaging unit 11. However, the present invention is not limited to this, and a part of this process may be performed by a device other than the drone 1, such as a server or a control terminal. Hereinafter, an example in which the server performs a part of the process of the control unit 20 will be described in detail.

[0071] FIG. 11 shows a configuration example of the drone 1A according to this modified example. FIG. 12 shows a configuration example of the server 40 according to this modified example. In this modified example, the processing performed by the map data generation unit 31, the action planning unit 32, and the route determination unit 33 in the drone 1 (FIG. 1) according to the above embodiment is performed by the server 40. The drone 1A has a communication unit 35A. The communication unit 35A is configured to communicate with the server 40 via wireless communication and the Internet. The server 40 has a communication unit 45, a map data generation unit 41, an action planning unit 42, and a route determination unit 43. The communication unit 45 is configured to communicate with the drone 1A via wireless communication and the Internet. The map data generation unit 41, the action planning unit 42, and the route determination unit 43 are the same as the map data generation unit 31, the action planning unit 32, and the route determination unit 33 according to the above embodiment.

[0072] With this configuration, the communication unit 35A of the drone 1A transmits the depth data DD and the position movement data DPM to the server 40, and the communication unit 45 of the server 40 receives the depth data DD and the position movement data DPM. The map data generation unit 41 generates map data MAP including an occupancy map based on the depth data DD and the position movement data DPM received by the communication unit 45. The action planning unit 42 plans the action of the drone 1 based on a preset destination or the control instruction data transmitted from the control terminal. The route determination unit 43 determines the flight route of the drone 1A based on the map data MAP and the determination result of the action planning unit 42. The communication unit 45 transmits data about the flight route of the drone 1A to the drone 1A, and the communication unit 35A of the drone 1A receives the data about the flight route. The airframe control unit 34 controls the airframe of the drone 1A based on the data about the flight route of the drone 1A received by the communication unit 35A.

[0073] [Other Modified Examples] Also, two or more of these modified examples may be combined.

[0074] <2. Second Embodiment> Next, the drone 2 according to the second embodiment will be described. In this embodiment, in addition to the six angles θ, the exposure frequency is set based on the environment around the drone. Note that the same reference numerals are given to substantially the same components as those of the drone 1 according to the first embodiment described above, and the description thereof will be omitted as appropriate.

[0075] FIG. 13 shows a configuration example of the drone 2. The drone 2 includes a control unit 50. The control unit 50 has a map data generation unit 51 and an exposure frequency setting unit 56.

[0076] The map data generation unit 51 is configured to generate map data MAP including an occupancy map based on the depth data DD and the position movement data DPM, in the same manner as the map data generation unit 31 according to the first embodiment described above. Further, the map data generation unit 51 checks the size of the space around the drone 2 based on the map data MAP, and supplies data about the size of the space to the exposure frequency setting unit 56 as environment data DE.

[0077] The exposure frequency setting unit 56 is configured to set the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM, in the same manner as the exposure frequency setting unit 26 according to the first embodiment described above. Further, the exposure frequency setting unit 56 also performs a process of correcting the set exposure frequency based on the environment data DE supplied from the map data generation unit 51. Then, the exposure frequency setting unit 56 is configured to set the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency.

[0078] Figure 14 shows an example of the operation of the drone 2. In this example, the drone 2 flies in a narrow space in the vertical direction. In this case, around the drone 2 flying in the flight direction DIR, the space in the vertical direction is narrow and there is space in the left - right direction. When there is space in the left - right direction like this, since the drone 2 can move in this left - right direction, for example, when an obstacle appears, it can move in the left - right direction to avoid the obstacle. Therefore, in order to grasp the environment in the left - right direction more accurately, the exposure frequency setting unit 56 increases the exposure frequency in the left - right direction and decreases the exposure frequency in the vertical direction. Specifically, for example, when the exposure frequency in the flight direction DIR is FF, the exposure frequency in the direction opposite to the flight direction DIR is FB, the exposure frequency in the upward direction is FU, the exposure frequency in the downward direction is FD, the exposure frequency in the left direction is FL, and the exposure frequency in the right direction is FR, the exposure frequency setting unit 56 can set the exposure frequencies FF, FB, FU, FD, FL, FR as follows. FF>FL = FR>FU = FD>FB

[0079] In this example, the case where the space in the vertical direction is narrow and there is space in the left - right direction has been described. However, for example, when the space in the left - right direction is narrow and there is space in the vertical direction, the exposure frequency setting unit 56 increases the exposure frequency in the vertical direction and decreases the exposure frequency in the left - right direction. Specifically, the exposure frequency setting unit 56 can set the exposure frequencies FF, FB, FU, FD, FL, FR as follows. FF>FU = FD>FL = FR>FB

[0080] On the other hand, for example, when the drone 2 is flying in a space with nothing, there is space in all directions around the drone 2. In such a case, since the drone 2 can move in all directions, for example, when an obstacle appears, it can move in any direction to avoid the obstacle. Therefore, the exposure frequency setting unit 56 makes the exposure frequency in the left - right direction and the exposure frequency in the vertical direction about the same. Specifically, the exposure frequency setting unit 56 can set the exposure frequencies FF, FB, FU, FD, FL, FR as follows. FF > FL = FR = FU = FD > FB

[0081] Here, the depth data generation unit 24 and the map data generation unit 51 correspond to a specific example of the "environmental information generation unit" in the present disclosure. The map data MAP corresponds to a specific example of the "environmental information" in the present disclosure.

[0082] FIG. 15 shows an operation example of the control unit 50. This flowchart is obtained by adding steps S205 to S207 to the flowchart (FIG. 4) of the control unit 20 according to the above first embodiment.

[0083] Similar to the case of the above first embodiment, the position movement estimation unit 25 calculates the velocity vector VS of the drone 2 based on the image data DT, the position data DP, and the inertial data DI (step S101). Then, the exposure frequency setting unit 56 checks whether the velocity value indicated by the velocity vector VS is equal to or less than the threshold value TH1 (step S102).

[0084] In step S102, when the velocity value is not less than the threshold value TH1 ( "N" in step S102), the exposure frequency setting unit 56 calculates the angle θ between the flight direction DIR of the drone 2 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103). Then, the exposure frequency setting unit 56 sets the exposure frequency of each of the six stereo cameras 100 based on the six angles θ (step S104).

[0085] Next, the map data generation unit 51 checks the space in the vertical and horizontal directions based on the map data MAP (step S205).

[0086] Next, the exposure frequency setting unit 56 checks whether the space in the vertical direction or the horizontal direction is narrow (step S206). When there is space in both the vertical and horizontal directions or when the space in both the vertical and horizontal directions is narrow ( "N" in step S206), the process proceeds to step S106.

[0087] In step S206, when the space in the vertical direction or the horizontal direction is narrow (\"Y\" in step S206), the exposure frequency setting unit 56 corrects the exposure frequency of each of the six stereo cameras 100 set in step S104 (step S207). For example, when the space in the vertical direction is narrow and there is space in the horizontal direction, the exposure frequency setting unit 56 increases the exposure frequency in the horizontal direction and decreases the exposure frequency in the vertical direction. Also, for example, when the space in the horizontal direction is narrow and there is space in the vertical direction, the exposure frequency setting unit 56 increases the exposure frequency in the vertical direction and decreases the exposure frequency in the horizontal direction. Then, the exposure frequency setting unit 56 sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency. Then, the process proceeds to the process of step S106.

[0088] Thereafter, it is the same as in the case of the first embodiment described above.

[0089] In this way, in the drone 2, the exposure frequency of each of the six stereo cameras 100 is set based on the map data MAP indicating the environment around the drone 2. As a result, in the drone 2, for example, when the space in the vertical direction is narrow and there is space in the horizontal direction, the exposure frequency in the horizontal direction can be increased and the exposure frequency in the vertical direction can be decreased. Thereby, in the drone 2, the environment in the horizontal direction can be grasped more accurately, so that, for example, when an obstacle appears, it can move appropriately in the horizontal direction to avoid the obstacle. In this way, in the drone 2, since the arithmetic resources of the control unit 50 can be largely allocated to the arithmetic processing of the image data obtained by imaging in the horizontal direction, for example, the environment can be grasped more accurately while effectively using the limited arithmetic resources.

[0090] As described above, in the present embodiment, the exposure frequency of each of the six stereo cameras is set based on the map data indicating the environment around the drone, so that the computing resources can be effectively utilized. Other effects are the same as those in the case of the first embodiment described above.

[0091] [Modification Example 2-1] In the above embodiment, the exposure frequency is set based on the size of the space around the drone 2, but it is not limited to this. Instead, for example, by performing image analysis processing based on the captured image, it can be confirmed whether there is something that can be an obstacle when the drone 2 flies, and the exposure frequency can be set based on the confirmation result. Hereinafter, this modification example will be described in detail.

[0092] FIG. 16 shows a configuration example of the drone 2A according to this modification example. The drone 2A includes a control unit 50A. The control unit 50A has an image analysis unit 58A, a map data generation unit 51A, and an exposure frequency setting unit 56A.

[0093] The image analysis unit 58A is configured to analyze the imaging target by performing image analysis processing using a semantic segmentation method based on the image data DT. Then, the image analysis unit 58A supplies the analysis result to the map data generation unit 51A as segmentation data DS.

[0094] The map data generation unit 51A is configured to generate map data MAP2 including a semantic occupancy map based on depth data DD, position movement data DPM, and segmentation data DS. Specifically, the map data generation unit 31 generates the map data MAP2 by performing a voting process on each voxel in the semantic occupancy map based on the depth values included in the depth data DD and the information about the position of the drone 1 included in the position movement data DPM. The semantic occupancy map includes data about the correspondence between each voxel and the imaging target. Specifically, for example, each voxel is attached with data indicating the imaging target such as empty, ground, tree, etc. Then, based on the map data MAP2, the map data generation unit 51A checks whether there is anything that may become an obstacle when the drone 2 flies, and supplies the check result to the exposure frequency setting unit 56A as environmental data DE2. Specifically, for example, when there is a tree near the drone 2, there is a possibility that a bird may fly in or fly away, which may interfere with the flight. Also, for example, when there is a door near the drone 2, there is a possibility that the door may open and interfere with the flight of the drone 2. Therefore, based on the semantic occupancy map, the map data generation unit 51A checks whether there is a tree or a door nearby in this example, and supplies the check result to the exposure frequency setting unit 56A as environmental data DE2.

[0095] The exposure frequency setting unit 56A is configured to set the exposure frequencies of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM. Also, the exposure frequency setting unit 56A performs a process of correcting the set exposure frequencies based on the environmental data DE2 supplied from the map data generation unit 51. Specifically, when there is something that may become an obstacle when the drone 2 flies, the exposure frequency is corrected so as to increase the exposure frequency of the stereo camera 100 that images the potential obstacle. Then, the exposure frequency setting unit 56A is configured to set the respective exposure timings of the six stereo cameras 100 based on the set exposure frequencies.

[0096] Here, the image analysis unit 58A corresponds to a specific example of the "image analysis unit" in the present disclosure. The depth data generation unit 24 and the map data generation unit 51A correspond to specific examples of the "environmental information generation unit" in the present disclosure. The map data MAP2 corresponds to a specific example of the "environmental information" in the present disclosure.

[0097] With this configuration, in the drone 2A, the exposure frequency of the stereo camera 100 that captures objects that may become obstacles during flight can be increased, so that the environment can be grasped more accurately, and thus obstacles can be appropriately avoided.

[0098] [Modification Example 2-2] In the above embodiment, the control unit 50 of the drone 2 controls the flight of the drone 2 based on the data supplied from the imaging unit 11, the GPS receiving unit 12, the pressure sensor 13, and the inertial measurement unit 14, and also performs the process of controlling the operation of the imaging unit 11. However, the present invention is not limited to this, and a part of this process may be performed by a device other than the drone 2, such as a server or a control terminal. Hereinafter, an example in which the server performs a part of the process of the control unit 50 will be described in detail.

[0099] FIG. 17 shows a configuration example of the drone 2B according to this modification example. FIG. 18 shows a configuration example of the server 60 according to this modification example. In this modification example, the processes performed by the map data generation unit 31, the action planning unit 32, and the route determination unit 33 in the drone 2 (FIG. 13) according to the above embodiment are performed by the server 60. The drone 2B has a communication unit 35B. The communication unit 35B is configured to communicate with the server 60 via wireless communication and the Internet. The server 60 has a communication unit 65, a map data generation unit 61, an action planning unit 42, and a route determination unit 43. The communication unit 65 is configured to communicate with the drone 2B via wireless communication and the Internet. The map data generation unit 61, the action planning unit 42, and the route determination unit 43 are the same as the map data generation unit 51, the action planning unit 32, and the route determination unit 33 according to the above embodiment.

[0100] With this configuration, the communication unit 35B of the drone 2B transmits the depth data DD and the position movement data DPM to the server 60, and the communication unit 65 of the server 60 receives this depth data DD and the position movement data DPM. The map data generation unit 61 generates map data MAP including an occupancy map and generates environment data DE based on the depth data DD and the position movement data DPM received by the communication unit 65. The action planning unit 42 plans the action of the drone 2B based on a preset destination or the control instruction data transmitted from the control terminal. The route determination unit 43 determines the flight route of the drone 2B based on the map data MAP and the determination result of the action planning unit 42. The communication unit 65 transmits the environment data DE and the data about the flight route of the drone 2B to the drone 2B, and the communication unit 35B of the drone 2B receives the environment data DE and the data about this flight route. The exposure frequency setting unit 56 sets the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM, and corrects the set exposure frequency based on the environment data DE received by the communication unit 35B. The airframe control unit 34 controls the airframe of the drone 2B based on the data about the flight route of the drone 2B received by the communication unit 35B.

[0101] <3. Third Embodiment> Next, the drone 3 according to the third embodiment will be described. In this embodiment, in addition to the six angles θ, the exposure frequency is set based on the moving objects around the drone as well. Note that the same reference numerals are given to substantially the same components as those of the drone 1 according to the first embodiment described above, and the description thereof will be omitted as appropriate.

[0102] FIG. 19 shows a configuration example of the drone 3. The drone 3 includes a control unit 70. The control unit 70 has an object detection unit 78, a map data generation unit 71, a moving object information analysis unit 79, and an exposure frequency setting unit 76.

[0103] The object detection unit 78 is configured to detect an object that may be an obstacle based on the image data DT. Then, the object detection unit 78 supplies the detection result to the map data generation unit 71 as object data DO.

[0104] The map data generation unit 71 is configured to generate map data MAP including an occupancy map based on the depth data DD, the position movement data DPM, and the object data DO.

[0105] The mobile object information analysis unit 79 is configured to analyze the position and movement of the object detected by the object detection unit 78 with respect to the drone 3 based on the depth data DD, the position movement data DPM, and the object data DO. Then, the mobile object information analysis unit 79 supplies the analysis result to the exposure frequency setting unit 76 as mobile object data DM.

[0106] Similar to the exposure frequency setting unit 26 according to the first embodiment, the exposure frequency setting unit 76 is configured to set the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM. Further, the exposure frequency setting unit 76 corrects the set exposure frequency so that the exposure frequency of the stereo camera 100 that images the detected object increases when the detected object approaches the drone 3 based on the mobile object data DM supplied from the mobile object information analysis unit 79. Then, the exposure frequency setting unit 76 is configured to set the respective exposure timings of the six stereo cameras 100 based on the set exposure frequency.

[0107] FIG. 20 shows an example of an operation of the drone 3. In this example, a bird 99 is flying around the drone 3, and the bird 99 is approaching the drone 9. That is, the velocity vector V99 of the bird 99 with respect to the drone 3 is directed toward the drone 3.

[0108] The exposure frequency setting unit 76 calculates the angle φ between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101 of each of the six stereo cameras 100. Specifically, similar to the case of FIG. 5, the exposure frequency setting unit 76 calculates the angle φA between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101A of the stereo camera 100A, calculates the angle φB between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101B of the stereo camera 100B, calculates the angle φC between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101C of the stereo camera 100C, calculates the angle φD between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101D of the stereo camera 100D, calculates the angle φE between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101E of the stereo camera 100E, and calculates the angle φF between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101F of the stereo camera 100F. The angle φ is an angle smaller than 180 degrees.

[0109] Then, the exposure frequency setting unit 76 sets the exposure frequency so as to increase the exposure frequency of the stereo camera 100 corresponding to the smallest angle φ among the six angles φ. Specifically, for example, in the example of FIG. 14, when the exposure frequencies of the stereo cameras 100A to 100D are F100A to F100D, the exposure frequency setting unit 76 can set the exposure frequency F100 as follows. F100B>F100A>F100C>F100D

[0110] Here, the object detection unit 78 and the moving object information analysis unit 79 correspond to a specific example of the "moving object detection unit" in the present disclosure.

[0111] FIG. 21 shows an operation example of the control unit 70. This flowchart is obtained by adding steps S305 to S307 to the flowchart (FIG. 4) of the control unit 20 according to the above first embodiment.

[0112] Similar to the case of the first embodiment, the position movement estimation unit 25 calculates the velocity vector VS of the drone 3 based on the image data DT, the position data DP, and the inertial data DI (step S101). Then, the exposure frequency setting unit 76 checks whether the velocity value indicated by the velocity vector VS is equal to or less than the threshold value TH1 (step S102).

[0113] In step S102, when the velocity value is not equal to or less than the threshold value TH1 ("N" in step S102), the exposure frequency setting unit 76 calculates the angle θ between the flight direction DIR of the drone 3 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103). Then, the exposure frequency setting unit 76 sets the exposure frequency of each of the six stereo cameras 100 based on the six angles θ (step S104).

[0114] Next, the moving object information analysis unit 79 analyzes the position and movement of the object detected by the object detection unit 78 with respect to the drone 3 based on the depth data DD, the position movement data DPM, and the object data DO (step S305).

[0115] Next, the exposure frequency setting unit 76 checks whether there is an approaching object based on the analysis result of the moving object information analysis unit 79 (step S306). If there is no approaching object ("N" in step S306), the process proceeds to the process of step S106.

[0116] In step S306, when there is an approaching object (i.e., “Y” in step S306), the exposure frequency setting unit 76 corrects the respective exposure frequencies of the six stereo cameras 100 so as to increase the exposure frequency of the stereo camera 100 that captures the object among the six stereo cameras 100 (step S307). For example, the exposure frequency setting unit 76 calculates the angle φ between the direction opposite to the direction indicated by the velocity vector V99 and the respective optical axis directions 101 of the six stereo cameras 100, and sets the exposure frequency so as to increase the exposure frequency of the stereo camera 100 corresponding to the smallest angle φ among the six angles φ. Then, based on the set exposure frequency, the exposure frequency setting unit 76 sets the respective exposure timings of the six stereo cameras 100. Then, the process proceeds to the process of step S106.

[0117] Thereafter, it is the same as in the case of the first embodiment described above.

[0118] In this way, in the drone 3, based on the processing results of the object detection unit 78 and the moving object information analysis unit 79, the respective exposure frequencies of the six stereo cameras 100 are set. Thereby, in the drone 3, for example, when an object approaches, the exposure frequency of the stereo camera 100 that captures the object can be increased. Thereby, in the drone 3, the environment including the approaching object can be grasped more accurately, so that the object can be appropriately avoided. In this way, in the drone 3, the arithmetic resources of the control unit 70 can be largely allocated to the arithmetic processing of the image data obtained by, for example, capturing an approaching object, so that the limited arithmetic resources can be effectively utilized while the environment can be grasped more accurately.

[0119] As described above, in this embodiment, based on the processing results of the object detection unit and the moving object information analysis unit, the respective exposure frequencies of the six stereo cameras are set, so that the arithmetic resources can be effectively utilized. Other effects are the same as in the case of the first embodiment described above.

[0120] [Modification Example 3] Each modification example of the above first and second embodiments may be applied to the drone 3 according to the above embodiment.

[0121] <4. Fourth Embodiment> Next, the drone 4 according to the fourth embodiment will be described. In this embodiment, in addition to the six angles θ, the exposure frequency is set based on the overlapping rate of the image regions between the captured images obtained at mutually different exposure timings. Note that the same reference numerals are given to substantially the same components as those of the drone 1 according to the first embodiment, and the description thereof will be omitted as appropriate.

[0122] FIG. 22 shows a configuration example of the drone 4. The drone 4 includes a control unit 80. The control unit 80 has an exposure frequency setting unit 86.

[0123] The exposure frequency setting unit 86 is configured to set the exposure frequencies of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM, in the same manner as the exposure frequency setting unit 26 according to the first embodiment. Further, the exposure frequency setting unit 86 calculates the overlapping rate of the image regions in the captured images obtained at mutually different exposure timings for each of the six stereo cameras 100 based on the depth data DD and the position movement data DPM, and performs a process of correcting the set exposure frequency based on the overlapping rate. Then, the exposure frequency setting unit 86 is configured to set the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency.

[0124] FIG. 23 shows an example of an operation of the drone 4. In this example, the stereo camera 100F that images the downward direction of the drone 4 performs the exposure operation twice at different exposure timings. Since the drone 4 is moving, the image area W1 in the first exposure operation EX and the image area W2 in the second exposure operation EX do not coincide and are shifted. Therefore, a part of the image area W1 and a part of the image area W2 overlap each other as shown in FIG. 23. In particular, when the moving speed of the drone 4 is high or when the distance from the drone 4 to the imaging target is short, the overlapping image area becomes wider. The exposure frequency setting unit 86 calculates a duplication rate indicating the degree of duplication of the image area based on the moving speed of the drone 4 and the distance from the drone 4 to the imaging target. Then, when the duplication rate of the image area in the captured image captured by the stereo camera 100F is low, the exposure frequency setting unit 86 increases the exposure frequency of the stereo camera 100F in order to increase the duplication rate.

[0125] Here, the depth data generation unit 24 corresponds to a specific example of the "distance calculation unit" in the present disclosure.

[0126] FIG. 24 shows an example of an operation of the control unit 80. This flowchart is obtained by adding steps S305 to S307 to the flowchart (FIG. 4) of the control unit 20 according to the above first embodiment.

[0127] Similar to the case of the first embodiment described above, the position movement estimation unit 25 calculates the velocity vector VS of the drone 4 based on the image data DT, the position data DP, and the inertial data DI (step S101). Then, the exposure frequency setting unit 56 checks whether the speed value indicated by the velocity vector VS is equal to or less than the threshold value TH1 (step S102).

[0128] In step S102, when the speed value is not less than the threshold value TH1 (i.e., "N" in step S102), the exposure frequency setting unit 56 calculates the angle θ between the flight direction DIR of the drone 4 indicated by the speed vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103). Then, the exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100 based on the six angles θ (step S104).

[0129] Next, the exposure frequency setting unit 86 calculates the overlapping rate of the image regions in the captured images obtained at different exposure timings for each of the six stereo cameras 100 based on the moving speed of the drone 4 and the distance from the drone 4 to the imaging target (step S405). Specifically, the exposure frequency setting unit 86 calculates the overlapping rate for the stereo camera 100A, calculates the overlapping rate for the stereo camera 100B, calculates the overlapping rate for the stereo camera 100C, calculates the overlapping rate for the stereo camera 100D, calculates the overlapping rate for the stereo camera 100E, and calculates the overlapping rate for the stereo camera 100F.

[0130] Next, the exposure frequency setting unit 86 checks whether at least any one of the overlapping rates of the six stereo cameras 100 is less than or equal to the threshold value TH2 (step S406). If not all of the overlapping rates are less than or equal to the threshold value TH2 (i.e., "N" in step S406), the process proceeds to step S106.

[0131] In step S406, when at least any one of the overlapping rates of the six stereo cameras 100 is less than or equal to the threshold value TH2 (i.e., "Y" in step S406), the exposure frequency setting unit 86 corrects the exposure frequency of each of the six stereo cameras 100 so that the overlapping rate less than or equal to the threshold value TH2 exceeds the threshold value TH2 (step S407). Then, the exposure frequency setting unit 86 sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency. Then, the process proceeds to step S106.

[0132] Thereafter, it is the same as in the case of the first embodiment described above.

[0133] In this way, in the drone 4, based on the moving speed of the drone 4 and the distance from the drone 4 to the imaging target, the exposure frequency of each of the six stereo cameras 100 is set. As a result, in the drone 4, for example, when the overlapping rate of the image regions is low, the exposure frequency of the stereo camera 100 with a low overlapping rate can be increased. Thereby, in the drone 4, the overlapping rate of the image regions can be increased, so that the accuracy of the VIO process in the position movement estimation unit 25 can be increased, and thus the estimation accuracy of the position and movement of the drone 4 can be increased.

[0134] As described above, in the present embodiment, based on the moving speed of the drone and the distance from the drone to the imaging target, the exposure frequency of each of the six stereo cameras is set, so that the estimation accuracy of the position and movement of the drone can be increased. Other effects are the same as in the case of the first embodiment described above.

[0135] [Modification Example 4] Each modification example of the first and second embodiments may be applied to the drone 4 according to the above embodiment.

[0136] Although the present technology has been described by giving several embodiments and modification examples, the present technology is not limited to these embodiments and the like, and various modifications are possible.

[0137] For example, in each of the above embodiments, six stereo cameras 100 are provided, but the present invention is not limited to this, and instead, two or more and five or less stereo cameras 100 may be provided, or seven or more stereo cameras 100 may be provided.

[0138] For example, in each of the above embodiments, the GPS receiver 12, the pressure sensor 13, and the inertial measurement unit 14 are provided, but the present invention is not limited to this, and for example, some of these may be omitted.

[0139] Also, for example, in each of the above embodiments, the present technology is applied to a drone, but it is not limited thereto. Instead, for example, it may be applied to a robot that travels on planar information, for example.

[0140] Note that the effects described in this specification are merely illustrative and not limiting, and there may be other effects.

[0141] Note that the present technology can be configured as follows. According to the present technology having the following configuration, computing resources can be effectively utilized.

[0142] (1) A setting unit that calculates a camera angle between the moving direction of the own device and the optical axis direction of each of a plurality of stereo cameras, and sets the exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles; An exposure control unit that controls the operations of the plurality of stereo cameras based on the plurality of exposure frequencies set by the setting unit An exposure control device comprising: (2) When the moving speed of the own device is faster than a predetermined speed, the setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles The exposure control device according to (1) above. (3) The setting unit makes the exposure frequency of a first stereo camera having the smallest camera angle among the plurality of stereo cameras higher than the exposure frequency of a second stereo camera different from the first stereo camera among the plurality of stereo cameras The exposure control device according to (1) or (2) above. (4) The setting unit When the moving speed of the own device is a first speed, sets the exposure frequency of the first stereo camera to a first exposure frequency, When the moving speed of the own device is a second speed faster than the first speed, sets the exposure frequency of the first stereo camera to a second exposure frequency higher than the first exposure frequency The exposure control device according to (3) above. (5) The apparatus further includes an environmental information generation unit that generates environmental information around the own apparatus based on imaging results of the plurality of stereo cameras. The setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the environmental information in addition to the plurality of camera angles. The exposure control device according to any one of (1) to (4) above. (6) The setting unit detects the size of the space around based on the environmental information, and sets the exposure frequency of a third stereo camera that images the direction in which the space is wide among the plurality of stereo cameras to be higher than the exposure frequency of a fourth stereo camera that is different from the third stereo camera among the plurality of stereo cameras. The exposure control device according to (5) above. (7) The apparatus further includes an image analysis unit that analyzes an imaging object based on imaging results of the plurality of stereo cameras. The environmental information includes information about the imaging object. The exposure control device according to (5) above. (8) The apparatus further includes a moving object detection unit that detects a moving object around the own apparatus based on imaging results of the plurality of stereo cameras. The setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the detection result of the moving object detection unit in addition to the plurality of camera angles. The exposure control device according to any one of (1) to (4) above. (9) When the distance between the moving object and the own apparatus decreases, the setting unit sets the exposure frequency of a fifth stereo camera that images the moving object among the plurality of stereo cameras to be higher than the exposure frequency of a sixth stereo camera that is different from the fifth stereo camera among the plurality of stereo cameras. The exposure control device according to (8) above. (10) The apparatus further includes a distance calculation unit that calculates the distance to the imaging object based on imaging results of the plurality of stereo cameras. In addition to the plurality of camera angles, the setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the moving speed of the own device and the distance to the imaging target. The exposure control device according to any one of (1) to (4) above. (11) The setting unit calculates the overlapping rate of the image regions between two imaging images with different exposure timings in one of the plurality of stereo cameras based on the moving speed of the own device and the distance to the imaging target, and increases the exposure frequency of the one stereo camera when the overlapping rate is lower than a predetermined value. The exposure control device according to (10) above. (12) The setting unit sets the exposure timing of each of the plurality of stereo cameras based on the exposure frequency of each of the plurality of stereo cameras. The exposure control unit controls the operations of the plurality of stereo cameras based on the plurality of exposure timings set by the setting unit. The exposure control device according to any one of (1) to (11) above. (13) The setting unit sets the exposure timing of each of the plurality of stereo cameras so that the exposure timings of the plurality of stereo cameras are different from each other. The exposure control device according to (12) above. (14) The apparatus further includes an image acquisition unit that acquires imaging results of the plurality of stereo cameras via a single bus wiring connected to the plurality of stereo cameras. The image acquisition unit acquires the imaging results of the plurality of stereo cameras in different periods respectively. The exposure control device according to (12) or (13) above. (15) Each of the plurality of stereo cameras has a plurality of image sensors. The exposure timings of the plurality of image sensors are the same as each other. The image acquisition unit acquires the imaging results of the plurality of image sensors in different periods respectively. The exposure control device according to (14) above. Calculate a camera angle between a moving direction of the device and an optical axis direction of each of a plurality of stereo cameras, and set an exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles; Control operations of the plurality of stereo cameras based on the set plurality of exposure frequencies; An exposure control method including the above.

[0143] This application claims priority based on Japanese Patent Application No. 2020-011031 filed with the Japan Patent Office on January 27, 2020, and all contents of this application are incorporated herein by reference.

[0144] Those skilled in the art can conceive various modifications, combinations, sub-combinations, and changes according to design requirements and other factors, but it is understood that they are included in the scope of the appended claims and their equivalents.

Claims

Claims 1. A setting unit that calculates a camera angle between a moving direction of its own device provided on a moving body and an optical axis direction of each of a plurality of stereo cameras provided on the moving body so that the optical axis directions are different from each other, and sets an exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles; An exposure control unit that controls operations of the plurality of stereo cameras based on the plurality of exposure frequencies set by the setting unit An imaging control device comprising: Claims 2. When the moving speed of its own device is higher than a predetermined speed, the setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles. The imaging control device according to claim 1. Claims 3. The setting unit sets the exposure frequency of a first stereo camera having the smallest camera angle among the plurality of stereo cameras to be higher than the exposure frequency of a second stereo camera different from the first stereo camera among the plurality of stereo cameras. The imaging control device according to claim 1. Claims 4. The setting unit When the moving speed of its own device is a first speed, sets the exposure frequency of the first stereo camera to a first exposure frequency; When the moving speed of its own device is a second speed higher than the first speed, sets the exposure frequency of the first stereo camera to a second exposure frequency higher than the first exposure frequency. The imaging control device according to claim 3. Claims 5. Further comprising an environment information generation unit that generates environment information around its own device based on imaging results of the plurality of stereo cameras, The setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles and also based on the environment information. The imaging control device according to claim 1. Claims 6. The setting unit detects the width of the space around based on the environment information, and sets the exposure frequency of a third stereo camera that images the direction in which the space is wide among the plurality of stereo cameras to be higher than the exposure frequency of a fourth stereo camera different from the third stereo camera among the plurality of stereo cameras. The imaging control device according to claim 5. Claims 7. Further comprising an image analysis unit that analyzes an imaging target based on imaging results of the plurality of stereo cameras, The environment information includes information about the imaging target. The imaging control device according to claim 5. Claims 8. The apparatus further includes a moving body detection unit that detects a moving body around the own apparatus based on imaging results of the plurality of stereo cameras. The setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the detection result of the moving body detection unit in addition to the plurality of camera angles. The imaging control apparatus according to claim 1.

9. When the distance between the moving body and the own apparatus decreases, the setting unit increases the exposure frequency of a fifth stereo camera that images the moving body among the plurality of stereo cameras to be higher than the exposure frequency of a sixth stereo camera that is different from the fifth stereo camera among the plurality of stereo cameras. The imaging control apparatus according to claim 8.

10. The apparatus further includes a distance calculation unit that calculates a distance to an imaging target based on imaging results of the plurality of stereo cameras. The setting unit sets the exposure frequency of each of the plurality of stereo cameras based on the moving speed of the own apparatus and the distance to the imaging target in addition to the plurality of camera angles. The imaging control apparatus according to claim 1.

11. The setting unit calculates an overlapping rate of an image area between two imaging images with different exposure timings in one of the plurality of stereo cameras based on the moving speed of the own apparatus and the distance to the imaging target, and increases the exposure frequency of the one stereo camera when the overlapping rate is lower than a predetermined value. The imaging control apparatus according to claim 10.

12. The setting unit sets the exposure timing of each of the plurality of stereo cameras based on the exposure frequency of each of the plurality of stereo cameras. The exposure control unit controls operations of the plurality of stereo cameras based on the plurality of exposure timings set by the setting unit. The imaging control apparatus according to claim 1.

13. The setting unit sets the exposure timing of each of the plurality of stereo cameras such that the exposure timings of the plurality of stereo cameras are different from each other. The imaging control apparatus according to claim 12.

14. The apparatus further includes an image acquisition unit that acquires imaging results of the plurality of stereo cameras via a single bus wiring connected to the plurality of stereo cameras. The image acquisition unit acquires the imaging results of the plurality of stereo cameras in different periods, respectively. The imaging control apparatus according to claim 12.

15. Each of the plurality of stereo cameras has a plurality of image sensors, the exposure timings of the plurality of image sensors are the same as each other, and the image acquisition unit acquires the imaging results of the plurality of image sensors respectively during different periods. The imaging control device according to claim 14.

16. A method for controlling imaging, comprising calculating a camera angle between a moving direction of a self-device provided on a moving body and an optical axis direction of each of a plurality of stereo cameras provided on the moving body so that the optical axis directions are different from each other, and setting an exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles; and controlling operations of the plurality of stereo cameras based on the set plurality of exposure frequencies. ​

Images