Imaging control apparatus and imaging control method

Abstract

The exposure control apparatus of the present disclosure includes a setting means that calculates a camera angle between a moving direction of the own apparatus and an optical axis direction of each of a plurality of stereo cameras, and sets an exposure frequency of a plurality of corresponding stereo cameras based on the camera angle; and an exposure control unit that controls operation of the plurality of stereo cameras based on the exposure frequency set by the setting means.

Description

Technical Field

[0001] This disclosure relates to an imaging control device for controlling the operation of an imaging apparatus, and an imaging control method used in such an imaging control device. Background Technology

[0002] Robotic devices typically include imaging equipment and generate map data indicating the environment surrounding the robotic device based on images acquired by the imaging equipment. For example, PTL 1 discloses a robotic device that generates an environmental map and uses the generated environmental map to determine actions.

[0003] Citation List

[0004] Patent documents

[0005] PTL 1: Japanese Unexamined Patent Application Publication Number: 2003-266349 Summary of the Invention

[0006] Incidentally, the goal is to be able to efficiently utilize computing resources when performing computational processing based on images acquired by imaging devices.

[0007] We hope to provide imaging control equipment and methods that can effectively utilize computing resources.

[0008] An imaging control device according to an embodiment of this disclosure includes a setting component and an exposure controller. The setting component is configured to calculate a camera angle between a movement direction of the device including a plurality of stereo cameras and the optical axis direction of each of the plurality of stereo cameras, and to set an exposure frequency for each of the plurality of stereo cameras based on the plurality of camera angles. The exposure controller is configured to control the operation of the plurality of stereo cameras based on the plurality of exposure frequencies set by the setting component.

[0009] An imaging control method according to an embodiment of the present disclosure includes: calculating a camera angle between the moving direction of a device including a plurality of stereo cameras and the optical axis direction of each of the plurality of stereo cameras, setting an exposure frequency of each of the plurality of stereo cameras based on the plurality of camera angles, and controlling the operation of the plurality of stereo cameras based on the set plurality of exposure frequencies.

[0010] In the imaging control device and imaging control method according to embodiments of the present disclosure, the camera angle between the moving direction of the imaging control device and the optical axis direction of each of a plurality of stereo cameras is calculated, and the exposure frequency of each of the plurality of stereo cameras is set based on the plurality of camera angles. Then, the operation of the plurality of stereo cameras is controlled based on the set plurality of exposure frequencies. Attached Figure Description

[0011] Figure 1This is a block diagram illustrating a configuration example of a drone according to a first embodiment of the present disclosure.

[0012] Figure 2 This is an illustrative diagram showing an example configuration of a drone according to an embodiment.

[0013] Figure 3 It is shown Figure 1 A block diagram illustrating an example configuration of the imaging component.

[0014] Figure 4 It is shown Figure 1 The flowchart shown is an example of how a drone can be operated.

[0015] Figure 5 It is shown Figure 1 The diagram illustrates an example of how a drone can be operated.

[0016] Figure 6 It is shown Figure 1 A timing diagram showing an example of the operation of the imaging component.

[0017] Figure 7 This is a timing diagram illustrating an operational example of the imaging component based on a comparative example.

[0018] Figure 8 It is shown Figure 1 The timing diagram shows an example of drone operation.

[0019] Figure 9 This is a timing diagram illustrating an operational example of a drone according to a modified example of the first embodiment.

[0020] Figure 10 This is a timing diagram illustrating an operational example of a drone according to another modified example of the first embodiment.

[0021] Figure 11 This is a block diagram illustrating a configuration example of a drone according to another modified example of the first embodiment.

[0022] Figure 12 This is a block diagram illustrating a configuration example of a server according to another modified example of the first embodiment.

[0023] Figure 13 This is a block diagram illustrating an example configuration of a drone according to a second embodiment.

[0024] Figure 14 It is shown Figure 13 The diagram illustrates an example of how a drone can be operated.

[0025] Figure 15 It is shown Figure 13 The flowchart shown is an example of how a drone can be operated.

[0026] Figure 16 This is a block diagram illustrating a configuration example of a drone according to a modified example of the second embodiment.

[0027] Figure 17 This is a block diagram illustrating a configuration example of a drone according to another modified example of the second embodiment.

[0028] Figure 18 This is a block diagram illustrating a configuration example of a server according to another modified example of the second embodiment.

[0029] Figure 19 This is a block diagram illustrating a configuration example of a drone according to a third embodiment.

[0030] Figure 20 It is shown Figure 19 The diagram illustrates an example of how a drone can be operated.

[0031] Figure 21 It is shown Figure 19 The flowchart shown is an example of how a drone can be operated.

[0032] Figure 22 This is a block diagram illustrating a configuration example of a drone according to a third embodiment.

[0033] Figure 23 It is shown Figure 22 The diagram illustrates an example of how a drone can be operated.

[0034] Figure 24 It is shown Figure 22 The flowchart shown is an example of how a drone can be operated. Detailed Implementation

[0035] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the description is given in the following order.

[0036] 1. First Embodiment

[0037] 2. Second Embodiment

[0038] 3. Third embodiment

[0039] 4. Fourth Embodiment

[0040] <1. First Embodiment>

[0041] [Configuration Example]

[0042] Figure 1 An example configuration of a drone 1 including an imaging control device according to an embodiment is shown. Figure 2 An example of the appearance of drone 1 is shown, in which Figure 2 (A) shows a floor plan. Figure 2 (B) shows a front view. It should be noted that the imaging control method according to one embodiment of this disclosure is implemented in this embodiment and is described together with it.

[0043] like Figure 1 As shown, the UAV 1 includes an imaging unit 11, a GPS (Global Positioning System) receiver 12, a barometric pressure sensor 13, an inertial measurement unit 14, and a controller 20. Furthermore, as... Figure 2 As shown, the UAV 1 includes a main body component 91, four arm components 92, and four rotor blades 93. An imaging component 11, a GPS receiver 12, a barometric pressure sensor 13, an inertial measurement unit 14, and a controller 20 are contained in the main body component 91 of the UAV 1.

[0044] Imaging component 11 ( Figure 1 It is configured to generate image data by imaging the surrounding environment of the UAV 1 based on instructions from the controller 20. For example... Figure 2 As shown, in this example, the imaging unit 11 includes six stereo cameras 100 (stereo cameras 100A to 100F). In this example, each of the six stereo cameras 100 includes two image sensors arranged at a predetermined distance from each other and is configured to generate image data (stereo image data) about two captured images having parallax between them by performing an exposure operation. The stereo image data includes a timestamp indicating the exposure timing. It should be noted that in this example, each of the six stereo cameras 100 includes two image sensors; however, each of the six stereo cameras 100 is not limited to this and may include, for example, three or more image sensors.

[0045] like Figure 2 As shown, six stereo cameras 100 are arranged in the main body component 91 such that their optical axis directions 101 are different from each other. Specifically, the optical axis direction 101A of stereo camera 100A is the X-axis direction; the optical axis direction 101B of stereo camera 100B is the Y-axis direction; the optical axis direction 101C of stereo camera 100C is opposite to the X-axis direction; the optical axis direction 101D of stereo camera 100D is opposite to the Y-axis direction; the optical axis direction 101E of stereo camera 100E is the Z-axis direction; and the optical axis direction 101F of stereo camera 100F is opposite to the Z-axis direction. With this configuration, stereo cameras 100A to 100D each perform imaging in the horizontal direction of the drone 1; stereo camera 100E performs imaging in the upward direction of the drone 1; and stereo camera 100F performs imaging in the downward direction of the drone 1.

[0046] Figure 3An example configuration of the imaging unit 11 is shown. The imaging unit 11 includes six stereo cameras 100 and bus wiring 109. The six stereo cameras 100 are coupled to the bus wiring 109. The bus wiring 109 includes, for example, multiple wirings. For example, MIPI (Mobile Industry Processor Interface) is suitable as the communication interface for the six stereo cameras 100. The six stereo cameras 100 transmit the generated image data to the controller 20 in a time-division manner via the bus wiring 109.

[0047] GPS receiver 12 is configured to detect the position of drone 1 by receiving signals transmitted from multiple artificial satellites. Barometric pressure sensor 13 is configured to detect air pressure. Information about the detected air pressure is used to detect the altitude of drone 1. Therefore, GPS receiver 12 and barometric pressure sensor 13 serve as position sensors for detecting the position of drone 1.

[0048] The inertial measurement unit 14 is configured using, for example, an IMU (Inertial Measurement Unit) and is configured to detect angular velocity and acceleration. The IMU includes, for example, an accelerometer sensor, a gyroscope sensor, a magnetometer, etc.

[0049] The controller 20 is configured to control the flight of the UAV 1 based on data provided from the imaging unit 11, the GPS receiver 12, the barometric pressure sensor 13, and the inertial measurement unit 14. Furthermore, the controller 20 performs processing for controlling the operation of the imaging unit 11. The controller 20 includes an image data acquisition unit 21, a position data acquisition unit 22, an inertial data acquisition unit 23, a depth data generator 24, a position movement estimation unit 25, an exposure frequency setting unit 26, a timing controller 27, a map data generator 31, an action planning unit 32, a route determination unit 33, and a fuselage controller 34. The controller 20 is configured using, for example, one or more processors.

[0050] The image data acquisition unit 21 is configured to acquire image data provided from the imaging unit 11 and provide the acquired data as image data DT to the depth data generator 24 and the position movement estimation unit 25.

[0051] The location data acquisition unit 22 is configured to acquire the detection results of the GPS receiver unit 12 and the barometric pressure sensor 13, and provide the acquired data as location data DP to the location movement estimation unit 25.

[0052] The inertial data acquisition unit 23 is configured to acquire the detection results of the inertial measurement unit 14 and provide the acquired data as inertial data DI to the position movement estimation unit 25.

[0053] The depth data generator 24 is configured to generate depth data DD based on stereo image data included in the image data DT, the depth data DD including a depth map indicating depth values ​​in the captured image.

[0054] The position and movement estimation unit 25 estimates the position and movement of the UAV 1 based on image data DT, position data DP, and inertial data DI. Specifically, the position and movement estimation unit 25 performs VIO (Visual Inertial Odometry) processing based on image data DT, and uses position data DP and inertial data DI to calculate the position, attitude, velocity vector VS, and angular velocity of the UAV 1 at the time indicated by the timestamp included in image data DT, and generates position and movement data DPM containing these data.

[0055] The exposure frequency setting component 26 is configured to set the exposure frequency of the six stereo cameras 100 in the imaging component 11 based on the velocity vector VS included in the position movement data DPM, and to set the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency. Specifically, the exposure frequency setting component 26 sets the exposure frequency of the six stereo cameras 100 based on the velocity value indicated by the velocity vector VS and the angle θ between the flight direction DIR of the UAV 1 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100. Then, the exposure frequency setting component 26 sets the exposure timing of the six stereo cameras 100 such that the exposure timings of the six stereo cameras 100 are different from each other.

[0056] The timing controller 27 is configured to control the operation timing of the six stereo cameras 100 based on instructions from the exposure frequency setting component 26.

[0057] Map data generator 31 is configured to generate a map data MAP including an occupancy map based on depth data DD and location movement data DPM. Specifically, map data generator 31 generates map data MAP by performing a voting process on each voxel in the occupancy map based on the depth values ​​included in the depth data DD and the position of UAV 1 included in the location movement data DPM.

[0058] Action planning component 32 is configured to plan the actions of UAV 1 based on a preset destination and control instruction data sent from the control terminal.

[0059] The route determination component 33 is configured to determine the flight path of the UAV 1 based on the map data MAP and the determination results of the action planning component 32.

[0060] The fuselage controller 34 is configured to control the fuselage of the UAV 1 based on the flight path determined by the route determination component 33. For example... Figure 2 As shown, each of the four rotor blades 93 is attached to the end of a corresponding one of the four arm components 92 and is configured to rotate. The four rotor blades 93 rotate based on the driving force generated by four motors (not shown). The fuselage controller 34 controls the operation of these four motors based on a defined flight path. This enables the UAV 1 to fly along a defined flight path.

[0061] Here, the exposure frequency setting component 26 corresponds to a specific example of the "setting component" in this disclosure. The timing controller 27 corresponds to a specific example of the "exposure controller" in this disclosure. The six stereo cameras 100 correspond to a specific example of the "multiple stereo cameras" in this disclosure. The angle θ corresponds to a specific example of the "camera angle" in this disclosure. The bus wiring 109 corresponds to a specific example of the "bus wiring" in this disclosure. The image data acquisition component 21 corresponds to a specific example of the "image acquisition component" in this disclosure.

[0062] [Operation and Work]

[0063] Next, the operation and function of the drone 1 according to this embodiment will be described.

[0064] (Overview of the overall operation)

[0065] First, refer to Figure 1 An overview of the overall operation of the UAV 1 is provided. The imaging unit 11 generates image data by imaging the environment surrounding the UAV 1, based on instructions from the controller 20. The GPS receiver 12 receives signals from multiple satellites to detect the position of the UAV 1. The barometric pressure sensor 13 detects air pressure. The inertial measurement unit 14 detects angular velocity and acceleration. The controller 20 controls the flight of the UAV 1 and the operation of the imaging unit 11 based on data provided from the imaging unit 11, the GPS receiver 12, the barometric pressure sensor 13, and the inertial measurement unit 14.

[0066] (Detailed instructions)

[0067] Figure 4An operational example of controller 20 is shown. In controller 20, depth data generator 24 generates depth data DD based on stereo image data included in image data DT, and position movement estimation unit 25 estimates the position and movement of UAV 1 based on image data DT, position data DP, and inertial data DI to generate position movement data DPM. Exposure frequency setting unit 26 sets the exposure frequency of the six stereo cameras 100 in imaging unit 11 based on the velocity vector VS included in position movement data DPM, and sets the individual exposure timing of the six stereo cameras 100 based on the set exposure frequency. Map data generator 31 generates map data MAP including an occupied map based on depth data DD and position movement data DPM. This operation is described in detail below.

[0068] The position movement estimation unit 25 calculates the velocity vector VS of the UAV 1 based on image data DT, position data DP, and inertial data DI (step S101). Specifically, the position movement estimation unit 25 performs VIO processing based on image data DT and calculates the velocity vector VS of the UAV 1 using position data DP and inertial data DI.

[0069] Next, the exposure frequency setting component 26 confirms whether the speed value indicated by the speed vector VS is equal to or less than the threshold TH1 (step S102).

[0070] In step S102, if the velocity value is not equal to or less than the threshold TH1 (N in step S102), the exposure frequency setting component 26 calculates the angle θ between the flight direction DIR of the UAV 1 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103).

[0071] Figure 5An example of angle θ is shown. In this example, the flight direction DIR is horizontal and is located close to the optical axis direction 101A between the optical axis directions 101A and 101B of stereoscopic camera 100A and stereoscopic camera 100B, respectively. The exposure frequency setting unit 26 calculates the angle θA between the flight direction DIR and the optical axis direction 101A of stereoscopic camera 100A; calculates the angle θB between the flight direction DIR and the optical axis direction 101B of stereoscopic camera 100B; calculates the angle θC between the flight direction DIR and the optical axis direction 101C of stereoscopic camera 100C; and calculates the angle θD between the flight direction DIR and the optical axis direction 101D of stereoscopic camera 100D. Furthermore, although not shown, the exposure frequency setting unit 26 calculates the angle θE between the flight direction DIR and the optical axis direction 101E of stereoscopic camera 100E, and calculates the angle θF between the flight direction DIR and the optical axis direction 101F of stereoscopic camera 100F. Angle θ is an angle less than 180 degrees.

[0072] In this example, the flight direction DIR is close to the optical axis direction 101A of the stereo camera 100A; therefore, angle θA is the smallest and angle θC is the largest. The flight direction DIR is horizontal; therefore, angles θE and θF are both 90 degrees.

[0073] Next, the exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100 based on six angles θ (step S104). Specifically, the exposure frequency setting unit 26 sets the exposure frequency so that the exposure frequency of the stereo camera 100 corresponding to the smallest angle θ among the six angles θ is higher than the exposure frequency of one of the other stereo cameras 100. That is, in order to more accurately grasp the environment on the flight direction DIR, the exposure frequency setting unit 26 sets the exposure frequency to increase the exposure frequency of the stereo cameras 100 performing imaging on the flight direction DIR.

[0074] For example, the exposure frequency setting unit 26 can calculate the cosine values ​​of the six angles θ and set the exposure frequency of each of the six stereo cameras 100 based on the cosine values. For example, when the flight direction DIR is the same as the optical axis direction 101A of the stereo camera 100A, the angles θA to θF are as follows.

[0075] Stereo camera 100A: θA = 0 degrees

[0076] Stereo camera 100B: θB = 90 degrees

[0077] Stereo camera 100C: θC = 180 degrees

[0078] Stereo Camera 100D: θD = 90 degrees

[0079] Stereo camera 100E: θE = 90 degrees

[0080] Stereo camera 100F: θF = 90 degrees

[0081] Therefore, the cosine value is as follows.

[0082] Stereo camera 100A:cosθA=1

[0083] Stereo camera 100B:cosθB=0

[0084] Stereo camera 100C:cosθC=-1

[0085] Stereo camera 100D: cosθD=0

[0086] Stereo camera 100E:cosθE=0

[0087] Stereo camera 100F:cosθF=0

[0088] In this example, cosθC is the smallest; therefore, divide these cosine values ​​by the absolute value of cosθC respectively, and add 2 to each result of this division. Thus, we obtain the following values.

[0089] 3D camera 100A:3

[0090] 100B:2 Stereo Camera

[0091] 100C stereo camera:1

[0092] 100D 3D Camera:2

[0093] 100E:2 Stereo Camera

[0094] 100F:2 Stereo Camera

[0095] In this example, the square of each of these values ​​is taken, which allows the exposure frequencies of the six stereo cameras 100 to be set as follows.

[0096] 3D camera 100A:9

[0097] 3D camera 100B:4

[0098] 100C stereo camera:1

[0099] 100D Stereo Camera:4

[0100] 100E:4 Stereo Camera

[0101] 100F:4 Stereo Camera

[0102] In other words, in this example, the exposure frequency ratio of stereo cameras 100A, 100B, 100D, 100E, and 100F, and 100C can be set to 9:4:1. It's important to note that in this example, the square of the value is used; however, if it's desired to reduce the difference between exposure frequencies, the square of the value can be omitted. In this case, the exposure frequency ratio is 3:2:1. Furthermore, if it's desired to increase the difference between exposure frequencies, the cube of the value can be used. In this case, the exposure frequency ratio is 27:8:1.

[0103] It should be noted that the above method of setting the respective exposure frequencies of the six stereo cameras 100 based on six angles θ is an example, but it is not limiting and any of the various methods can be used.

[0104] In step S102, when the speed value is equal to or less than the threshold TH1 ("Y" in step S102), the exposure frequency setting unit 26 sets the exposure frequency of each of the six stereo cameras 100 to the same frequency (step S105). That is, in order to grasp the entire environment around the drone 1, the exposure frequency setting unit 26 sets the exposure frequency of all stereo cameras 100 to the same frequency.

[0105] Therefore, the exposure frequency setting component 26 sets the exposure frequency of each of the six stereo cameras 100. Then, the exposure frequency setting component 26 sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency. At this time, the exposure frequency setting component 26 sets the exposure timing of each of the six stereo cameras 100 so that the exposure timings of the six stereo cameras 100 are different from each other.

[0106] Next, the timing controller 27 controls the operation timing of the six stereo cameras 100 based on the instruction from the exposure frequency setting component 26 (step S106).

[0107] In addition, the depth data generator 24 generates depth data DD based on the stereo image data included in the image data DT (step S107).

[0108] Next, the position movement estimation component 25 calculates the position of the UAV 1 at the time indicated by the timestamp based on the timestamp included in the image data DT (step S108).

[0109] Then, the map data generator 31 performs a voting process on each voxel in the occupied map based on the location of the drone 1 obtained in step S108 and the depth value obtained from the stereo image data including timestamps (step S109). The map data MAP is updated in this way.

[0110] This process ends here. Controller 20 repeats this process in each frame period.

[0111] As described above, in UAV 1, the exposure frequencies of the six stereo cameras 100 are set based on six angles θ. This allows UAV 1 to effectively utilize the computing resources in controller 20. That is, a higher exposure frequency is desirable for a more accurate understanding of the environment surrounding UAV 1. However, this increases the computational burden on controller 20. In UAV 1, the individual exposure frequencies of the six stereo cameras 100 are set based on six angles θ. Therefore, for example, the exposure frequency of the stereo camera 100 performing imaging in the flight direction DIR can be increased, and the exposure frequency of the stereo camera 100 performing imaging in the direction opposite to the flight direction DIR can be decreased. As a result, in UAV 1, much of the computing resources of controller 20 can be allocated to the computational processing of image data obtained by performing imaging in the flight direction DIR, which enables a more accurate understanding of the environment in the flight direction DIR while effectively utilizing limited computing resources.

[0112] Furthermore, when the flight speed of the drone 1 is higher than a predetermined speed, the exposure frequencies of the six stereo cameras 100 are set based on six angles θ. This allows the drone 1 to effectively utilize the computational resources in the controller 20. That is, when the flight speed is slow, the movement of the drone 1 is small; therefore, even with a low exposure frequency, the environment around the drone 1 can be grasped with a certain degree of accuracy. In contrast, when the flight speed is fast, the movement of the drone 1 is large; therefore, a high exposure frequency is desired to grasp the environment around the drone 1. However, in this case, the computational burden on the controller 20 increases. In the drone 1, when the flight speed is fast, the exposure frequencies of the six stereo cameras 100 are set based on six angles θ. Therefore, in the drone 1, when the flight speed is fast, much of the computational resources of the controller 20 can be allocated to the computational processing of image data obtained by performing imaging in the flight direction DIR, which allows for the efficient use of limited computational resources.

[0113] (Regarding exposure timing)

[0114] The exposure frequency setting component 26 sets the exposure timing of each of the six stereo cameras 100 so that the exposure timings of the six stereo cameras 100 are different from each other. The six stereo cameras 100 generate image data by performing exposure operations based on instructions from the controller 20. The six stereo cameras 100 then send the generated image data to the controller 20 in a time-division manner via bus wiring 109. This operation is described in detail below.

[0115] Figure 6An operational example of the imaging unit 11 is shown. In this example, the exposure timing is set in the order of stereo camera 100A, stereo camera 100B, stereo camera 100C...

[0116] First, the two image sensors 100A1 and 100A2 of the stereo camera 100A perform an exposure operation EX based on the synchronization signal SYNC1 during the time period from timing t11 to timing t12. The stereo camera 100A then performs a transmission operation TX during the time period from timing t12 to timing t13 to send image data generated by image sensor 100A1 to controller 20 via bus wiring 109, and performs a transmission operation TX during the time period from timing t13 to timing t14 to send image data generated by image sensor 100A2 to controller 20 via bus wiring 109.

[0117] Similarly, the two image sensors 100B1 and 100B2 of the stereo camera 100B perform an exposure operation EX based on the synchronization signal SYNC2 during the time period from timing t21 to timing t22. The stereo camera 100B then performs a transmission operation TX during the time period from timing t22 to timing t23 to send the image data generated by the image sensor 100B1 to the controller 20 via the bus wiring 109, and performs a transmission operation TX during the time period from timing t23 to timing t24 to send the image data generated by the image sensor 100B2 to the controller 20 via the bus wiring 109.

[0118] Similarly, the two image sensors 100C1 and 100C2 of the stereo camera 100C perform an exposure operation EX based on the synchronization signal SYNC3 during the time period from timing t31 to timing t32. The stereo camera 100C then performs a transmission operation TX during the time period from timing t32 to timing t33 ​​to transmit the image data generated by the image sensor 100C1 to the controller 20 via the bus wiring 109, and performs a transmission operation TX during the time period from timing t33 ​​to timing t34 to transmit the image data generated by the image sensor 100C2 to the controller 20 via the bus wiring 109.

[0119] Therefore, the exposure timings of the six stereo cameras 100 are set differently from each other. Then, the six stereo cameras 100 send the generated image data to the controller 20 in a time-division manner via bus wiring 109.

[0120] Therefore, in the drone 1, the exposure timings of the six stereo cameras 100 are different from each other, which makes it possible to reduce latency. That is, for example, when the exposure timings of the six stereo cameras 100 are the same, such as... Figure 7As shown, for example, in stereo cameras 100B and 100C, a time td is generated from the exposure operation EX to the transmission operation TX, and the controller 20 may not be able to acquire image data in time. In contrast, in the UAV 1, the exposure timings of the six stereo cameras 100 are different from each other; therefore, for example, as... Figure 6 As shown, the transmit operation TX can be performed shortly after the exposure operation EX, enabling the controller 20 to acquire image data promptly. Therefore, in the UAV 1, for example, the accuracy of computational processing can be improved.

[0121] (Regarding exposure frequency)

[0122] The exposure frequency setting component 26 sets the respective exposure frequencies of the six stereo cameras 100 based on six angles θ. The six stereo cameras 100 generate image data by performing exposure operations based on instructions from the controller 20. The controller 20 performs processing based on the image data generated by the six stereo cameras 100.

[0123] Figure 8 An example of the operation of controller 20 is shown, where (A) through (F) represent the operation of stereo cameras 100A, 100B, 100D, 100E, 100F, and 100C, respectively; (G) represents the operation of position movement estimation unit 25; and (H) represents the operation of depth data generator 24 and map data generator 31. In this example, the ratio of the exposure frequency of stereo camera 100A, the exposure frequency of each of stereo cameras 100B, 100D, 100E, and 100F, and the exposure frequency of stereo camera 100C is 4:2:1.

[0124] In this example, such as Figure 8 As shown in (A) to (F), the imaging unit 11 performs exposure operations in the order of stereo cameras 100A, 100B, 100D, 100A, 100E, 100F... Therefore, stereo camera 100A performs exposure operations at a rate of eight times per 26 frames; stereo cameras 100B, 100D, 100E, and 100F perform exposure operations at a rate of four times per 26 frames; and stereo camera 100C performs exposure operations at a rate of twice per 26 frames.

[0125] The positional motion estimation component 25 performs processing based on the provided image data DT. That is, as... Figure 8As shown in (G), the position movement estimation unit 25 first performs processing based on image data generated by stereo camera 100A, then performs processing based on image data generated by stereo camera 100B; performs processing based on image data generated by stereo camera 100D; performs processing based on image data generated by stereo camera 100A; performs processing based on image data generated by stereo camera 100E; and performs processing based on image data generated by stereo camera 100F. The same process continues thereafter.

[0126] Depth data generator 24 and map data generator 31 perform processing based on the provided image data DT. That is, as... Figure 8 As shown in (H), the depth data generator 24 and the map data generator 31 first perform processing based on image data generated by stereo camera 100A, then on image data generated by stereo camera 100B; then on image data generated by stereo camera 100D; then on image data generated by stereo camera 100A; then on image data generated by stereo camera 100E; and finally on image data generated by stereo camera 100F. The process continues thereafter.

[0127] Therefore, in the UAV 1, each of the six stereo cameras 100 performs an exposure operation at substantially equal time intervals. Specifically, for example, stereo camera 100A performs an exposure operation at a rate of once every 3 to 4 frames; stereo cameras 100B, 100D, 100E, and 100F perform an exposure operation at a rate of once every 6 to 7 frames; and stereo camera 100C performs an exposure operation at a rate of once every 13 frames. This enables the UAV 1 to accurately grasp its surrounding environment.

[0128] [Effect]

[0129] As described above, in this embodiment, the exposure frequencies of the six stereo cameras are set based on six angles θ, which makes it possible to make effective use of computing resources.

[0130] In this embodiment, when the flight speed is faster than a predetermined speed, the exposure frequencies of the six stereo cameras are set based on six angles θ, which makes it possible to make effective use of computing resources.

[0131] In this embodiment, the exposure timings of the six stereo cameras are different from each other, which reduces latency and thus improves the accuracy of computational processing, for example.

[0132] [Modified Example 1-1]

[0133] In the above embodiment, the exposure frequency setting component 26 increases the exposure frequency of the stereo camera 100 corresponding to the smallest angle θ among the six angles θ. At this time, the exposure frequency setting component 26 can increase the exposure frequency as the velocity value indicated by the velocity vector VS increases. Specifically, for example, when the flight direction DIR is the same as the optical axis direction 101A of the stereo camera 100A, at slower speeds, the ratio of the exposure frequency of the stereo camera 100A, each exposure frequency of the stereo cameras 100B, 100D, 100E, and 100F, and the exposure frequency of the stereo camera 100C can be set to 3:2:1, and at faster speeds, the ratio of the exposure frequency of the stereo camera 100A, each exposure frequency of the stereo cameras 100B, 100D, 100E, and 100F, and the exposure frequency of the stereo camera 100C can be set to 9:4:1.

[0134] [Modified Example 1-2]

[0135] In the above embodiments, the depth data generator 24 and the map data generator 31 perform processing in each frame, but this is not limiting. Instead, for example, as... Figure 9 As shown, there may be frames where the depth data generator 24 and map data generator 31 do not perform processing. In this example, the position movement estimation component 25 performs processing in each frame. Furthermore, the depth data generator 24 and map data generator 31 do not perform processing at a rate of once every 6 to 7 frames. Additionally, for example, there may be frames where the position movement estimation component 25 does not perform processing. In this example, the depth data generator 24 and map data generator 31 perform processing in each frame.

[0136] [Modified Examples 1-3]

[0137] In the above embodiments, for example, such as Figure 8 As shown, the position movement estimation component 25, the depth data generator 24, and the map data generator 31 perform processing based on the same image data, but this is not limiting. Conversely, for example, as... Figure 10 As shown, the image data to be processed by the position movement estimation unit 25 and the image data to be processed by the depth data generator 24 and the map data generator 31 can be different from each other. Figure 10 (A) through (F) respectively indicate the operation of stereo cameras 100A, 100B, 100D, 100E, 100F, and 100C; (G) indicates the operation of the position movement estimation unit 25; and (H) indicates the operation of the depth data generator 24 and the map data generator 31. Figure 10In (A) to (F), the solid line represents the stereo camera 100 that generates image data to be processed by the depth data generator 24 and the map data generator 31, and the dashed line represents the stereo camera 100 that generates image data to be processed by the position motion estimation unit 25. In this example, the position motion estimation unit 25 performs VIO processing on the image data generated by the stereo camera 100F at a rate of approximately once every two frames, and the stereo camera 100F performs imaging in the downward direction of the UAV 1. For example, in the case where the exposure timings of the two stereo cameras 100 overlap, such as... Figure 7 As shown, the timing of the transmit operation TX is shifted. For example, when there is a margin in the amount of communication data in the bus wiring 109 (e.g., when the image size is small), the exposure timing of multiple stereo cameras 100 can overlap with each other in this way.

[0138] [Modified Examples 1-4]

[0139] In the above embodiments, the controller 20 of the UAV 1 controls the flight of the UAV 1 based on data provided from the imaging unit 11, the GPS receiving unit 12, the barometric pressure sensor 13, and the inertial measurement unit 14, and performs processing for controlling the operation of the imaging unit 11. However, this is not limiting, and devices other than the UAV 1 (such as servers and control terminals) can perform part of this processing. An example of a server performing part of the processing of the controller 20 is described in detail below.

[0140] Figure 11 A configuration example of UAV 1A according to this modified example is shown. Figure 12 A configuration example of server 40 according to this modified example is shown. In this modified example, server 40 performs operations by drone 1 according to the above embodiment. Figure 1 The processing is performed by the map data generator 31, action planning component 32, and route determination component 33 in the UAV 1A. The UAV 1A includes a communication component 35A. The communication component 35A is configured to communicate with the server 40 via wireless communication and the Internet. The server 40 includes a communication component 45, a map data generator 41, an action planning component 42, and a route determination component 43. The communication component 45 is configured to communicate with the UAV 1A via wireless communication and the Internet. The map data generator 41, action planning component 42, and route determination component 43 are similar to the map data generator 31, action planning component 32, and route determination component 33 according to the above embodiment.

[0141] In this configuration, the communication component 35A of the UAV 1A sends depth data (DD) and position movement data (DPM) to the server 40, and the communication component 45 of the server 40 receives the depth data (DD) and position movement data (DPM). The map data generator 41 generates map data (MAP) including the occupied map based on the depth data (DD) and position movement data (DPM) received by the communication component 45. The action planning component 42 plans the actions of the UAV 1A based on a preset destination and control instructions sent from the control terminal. The route determination component 43 determines the flight path of the UAV 1A based on the map data (MAP) and the determination result of the action planning component 42. The communication component 45 sends data about the flight path of the UAV 1A to the UAV 1A, and the communication component 35A of the UAV 1A receives the data about the flight path. The fuselage controller 34 controls the fuselage of the UAV 1A based on the flight path data received by the communication component 35A.

[0142] [Other modification examples]

[0143] Furthermore, two or more of these modified examples can be combined.

[0144] <2. Second Embodiment>

[0145] Next, the drone 2 according to the second embodiment will be described. In this embodiment, in addition to six angles θ, the exposure frequency is set based on the environment surrounding the drone. It should be noted that components substantially the same as those of the drone 1 according to the first embodiment described above are indicated by the same reference numerals, and their descriptions are appropriately omitted.

[0146] Figure 13 An example configuration of the drone 2 is shown. The drone 2 includes a controller 50. The controller 50 includes a map data generator 51 and an exposure frequency setting component 56.

[0147] Map data generator 51 is configured to generate map data MAP including the occupied map based on depth data DD and position movement data DPM, similar to map data generator 31 according to the first embodiment described above. Furthermore, map data generator 51 determines the width of the space surrounding the drone 2 based on the map data MAP and provides data regarding the width of this space as environmental data DE to exposure frequency setting component 56.

[0148] Similar to the exposure frequency setting unit 26 according to the first embodiment described above, the exposure frequency setting unit 56 is configured to set the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM. Furthermore, the exposure frequency setting unit 56 performs processing to correct the set exposure frequency based on the environmental data DE provided from the map data generator 51. Then, the exposure frequency setting unit 56 sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency.

[0149] Figure 14 An operational example of the drone 2 is shown. In this example, the drone 2 flies in a narrow space in the vertical direction. In this case, there is a narrow space in the vertical direction and a space in the horizontal direction around the drone 2 flying in the flight direction DIR. With space existing in the horizontal direction in this way, the drone 2 can move in the horizontal direction; therefore, in the event of an obstacle, for example, the drone 2 can move in the horizontal direction to avoid the obstacle. Therefore, in order to more accurately grasp the environment in the horizontal direction, the exposure frequency setting component 56 increases the exposure frequency in the horizontal direction and decreases the exposure frequency in the vertical direction. More specifically, for example, the exposure frequency setting component 56 can set the exposure frequencies FF, FB, FU, FD, FL, and FR as follows, where FF represents the exposure frequency in the flight direction DIR; FB represents the exposure frequency in the direction opposite to the flight direction DIR; FU represents the exposure frequency in the upward direction; FD represents the exposure frequency in the downward direction; FL represents the exposure frequency in the left direction; and FR represents the exposure frequency in the right direction.

[0150] FF>FL=FR>FU=FD>FB

[0151] In this example, a case is described where there is a narrow space in the vertical direction and a space in the horizontal direction; however, for example, in the case where there is a narrow space in the horizontal direction and a space in the vertical direction, the exposure frequency setting component 56 increases the exposure frequency in the vertical direction and decreases the exposure frequency in the horizontal direction. Specifically, the exposure frequency setting component 56 can set the exposure frequencies FF, FB, FU, FD, FL, and FR as follows.

[0152] FF>FU=FD>FL=FR>FB

[0153] In contrast, when the drone 2 is flying in an empty space, there is space in all directions around the drone 2. In this case, the drone 2 can move in all directions; therefore, in the event of an obstacle, for example, the drone 2 can move in any direction to avoid the obstacle. Therefore, the exposure frequency setting component 56 makes the exposure frequency in the left-right direction and the exposure frequency in the vertical direction substantially equal to each other. Specifically, the exposure frequency setting component 56 can set the exposure frequencies FF, FB, FU, FD, FL, and FR as follows.

[0154] FF>FL=FR=FU=FD>FB

[0155] Here, depth data generator 24 and map data generator 51 correspond to specific examples of the "environmental information generator" in this disclosure. Map data MAP corresponds to a specific example of the "environmental information" in this disclosure.

[0156] Figure 15 An operational example of controller 50 is shown. This flowchart is a flowchart of adding steps S205 to S207 to controller 20 according to the first embodiment described above. Figure 4 The flowchart is formed from the process of creating the flowchart.

[0157] Similar to the first embodiment described above, the position movement estimation unit 25 calculates the velocity vector VS of the UAV 2 based on image data DT, position data DP, and inertial data DI (step S101). Then, the exposure frequency setting unit 26 confirms whether the velocity value indicated by the velocity vector VS is equal to or less than the threshold TH1 (step S102).

[0158] In step S102, if the velocity value is not equal to or less than the threshold TH1 (N in step S102), the exposure frequency setting component 56 calculates the angle θ between the flight direction DIR of the UAV 2 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103). Then, the exposure frequency setting component 56 sets the respective exposure frequency of the six stereo cameras 100 based on the six angles θ (step S104).

[0159] Next, the map data generator 51 confirms the space in the vertical and horizontal directions based on the map data MAP (step S205).

[0160] Next, the exposure frequency setting component 56 checks whether the space in the vertical or horizontal direction is narrow (step S206). If there is space in both the vertical and horizontal directions, or if both the vertical and horizontal directions are narrow ("N" in step S206), the process proceeds to the processing in step S106.

[0161] In step S206, when the space is narrow in the vertical or horizontal direction ("Y" in step S206), the exposure frequency setting unit 56 corrects the exposure frequencies of the six stereo cameras 100 set in step S104 (step S207). For example, if there is narrow space in the vertical direction and space in the horizontal direction, the exposure frequency setting unit 56 increases the exposure frequency in the horizontal direction and decreases the exposure frequency in the vertical direction. Furthermore, for example, if there is narrow space in the horizontal direction and space in the vertical direction, the exposure frequency setting unit 56 increases the exposure frequency in the vertical direction and decreases the exposure frequency in the horizontal direction. Then, the exposure frequency setting unit 56 sets the exposure timing of the six stereo cameras 100 based on the set exposure frequencies. Then, the process proceeds to the processing in step S106.

[0162] The following steps are similar to those in the first embodiment described above.

[0163] Therefore, in UAV 2, the exposure frequencies of the six stereo cameras 100 are set based on map data (MAP) indicating the environment surrounding UAV 2. Thus, in UAV 2, for example, when there is narrow space in the vertical direction and space in the horizontal direction, the exposure frequency in the horizontal direction can be increased, while the exposure frequency in the vertical direction can be decreased. Therefore, in UAV 2, the environment in the horizontal direction can be grasped more accurately; therefore, for example, when obstacles are present, UAV 2 can be moved appropriately in the horizontal direction to avoid them. Therefore, for example, in UAV 2, many of the computing resources of the controller 50 can be allocated to the computational processing of image data obtained by imaging in the horizontal direction, enabling a more accurate grasp of the environment while efficiently utilizing limited computing resources.

[0164] As described above, in this embodiment, the exposure frequencies of the six stereo cameras are set based on map data indicating the environment surrounding the drone, which allows for efficient use of computing resources. Other effects are similar to those in the first embodiment described above.

[0165] [Modified Example 2-1]

[0166] In the above embodiment, the exposure frequency is set based on the width of the space surrounding the drone 2, but this is not limiting. Alternatively, for example, image analysis processing can be performed based on the captured images to determine if there are any potential obstacles to the drone 2's flight, and the exposure frequency can be set based on the result of this determination. This modified example is described in detail below.

[0167] Figure 16A configuration example of a drone 2A according to this modified example is shown. The drone 2A includes a controller 50A. The controller 50A includes an image analysis unit 58A, a map data generator 51A, and an exposure frequency setting unit 56A.

[0168] Image analysis unit 58A is configured to perform image analysis processing using semantic segmentation technology, analyzing the imaging object based on image data DT. Then, image analysis unit 58A provides the results of this analysis as segmentation data DS to map data generator 51A.

[0169] Map data generator 51A is configured to generate map data MAP2, including a semantic occupancy map, based on depth data DD, position movement data DPM, and segmentation data DS. Specifically, map data generator 31 performs a voting process on each voxel in the semantic occupancy map based on information about the depth values ​​included in the depth data DD and the position of UAV 1 included in the position movement data DPM to generate map data MAP2. The semantic occupancy map includes data about the correspondence between each voxel and the imaged object. Specifically, for example, data indicating the imaged object (e.g., sky, ground, and trees) is appended to each voxel. Then, map data generator 51A confirms the presence of possible obstacles to the flight of UAV 2 based on map data MAP2 and provides the result of this confirmation as environmental data DE2 to exposure frequency setting component 56A. Specifically, for example, if there is a tree near UAV 2, birds may fly over or fly away, becoming obstacles to flight. Furthermore, for example, if there is a door near UAV 2, the door may be open, becoming an obstacle to the flight of UAV 2. Therefore, in this example, the map data generator 51A determines whether there is a tree or door nearby based on the semantic occupancy map, and provides the result of this determination as environmental data DE2 to the exposure frequency setting component 56A.

[0170] The exposure frequency setting unit 56A is configured to set the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM. Furthermore, the exposure frequency setting unit 56A also performs processing to correct the set exposure frequency based on environmental data DE2 provided from the map data generator 51. Specifically, in the event of potential obstacles during the flight of the UAV 2, the exposure frequency is corrected to increase the exposure frequency of the stereo cameras 100 that perform imaging of the potential obstacles. Then, the exposure frequency setting unit 56A sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency.

[0171] Here, image analysis component 58A corresponds to a specific example of "image analysis component" in this disclosure. Depth data generator 24 and map data generator 51A correspond to specific examples of "environmental information generator" in this disclosure. Map data MAP2 corresponds to a specific example of "environmental information" in this disclosure.

[0172] With this configuration, the exposure frequency of the stereo camera 100 in the UAV 2A, which images potential flying obstacles, can be increased. This allows for a more accurate understanding of the environment, enabling the appropriate avoidance of obstacles.

[0173] [Modified Example 2-2]

[0174] In the above embodiments, the controller 50 of the UAV 2 controls the flight of the UAV 2 based on data provided from the imaging unit 11, the GPS receiving unit 12, the barometric pressure sensor 13, and the inertial measurement unit 14, and performs processing for controlling the operation of the imaging unit 11. However, this is not limiting, and devices other than the UAV 2 (such as servers and control terminals) can perform part of this processing. An example of a server performing part of the processing of the controller 50 is described in detail below.

[0175] Figure 17 A configuration example of UAV 2B according to this modified example is shown. Figure 18 A configuration example of server 60 according to this modified example is shown. In this modified example, server 60 performs operations by drone 2 according to the above embodiment. Figure 13 The processing is performed by the map data generator 31, action planning component 32, and route determination component 33 in the UAV 2B. The UAV 2B includes a communication component 35B. The communication component 35B is configured to communicate with the server 60 via wireless communication and the Internet. The server 60 includes a communication component 65, a map data generator 61, an action planning component 42, and a route determination component 43. The communication component 65 is configured to communicate with the UAV 2B via wireless communication and the Internet. The map data generator 61, action planning component 42, and route determination component 43 are similar to the map data generator 51, action planning component 32, and route determination component 33 according to the above embodiment.

[0176] In this configuration, the communication component 35B of the UAV 2B sends depth data DD and position movement data DPM to the server 60, and the communication component 65 of the server 60 receives the depth data DD and position movement data DPM. The map data generator 61 generates map data MAP, including the occupied map, based on the depth data DD and position movement data DPM received by the communication component 65, and generates environmental data DE. The action planning component 42 plans the actions of the UAV 2B based on a preset destination and control instructions sent from the control terminal. The route determination component 43 determines the flight path of the UAV 2B based on the map data MAP and the determination results of the action planning component 42. The communication component 65 sends the environmental data DE and data regarding the flight path of the UAV 2B to the UAV 2B, and the communication component 35B of the UAV 2B receives the environmental data DE and the data regarding the flight path. The exposure frequency setting component 56 sets the exposure frequency of the six stereo cameras 100 in the imaging component 11 based on the velocity vector VS included in the position movement data DPM, and corrects the set exposure frequency based on the environmental data DE received by the communication component 35B. The fuselage controller 34 controls the fuselage of the UAV 2BA based on the flight path data received by the communication component 35B.

[0177] <3. Third Embodiment>

[0178] Next, the drone 3 according to the third embodiment will be described. In this embodiment, in addition to the six angles θ, the exposure frequency is also set based on the moving bodies around the drone. It should be noted that components substantially the same as those of the drone 1 according to the first embodiment described above are indicated by the same reference numerals, and their descriptions are appropriately omitted.

[0179] Figure 19 An example configuration of UAV 3 is shown. UAV 3 includes a controller 70. Controller 70 includes an object detector 78, a map data generator 71, a moving object information analysis component 79, and an exposure frequency setting component 76.

[0180] Object detector 78 is configured to detect objects that may be obstacles based on image data DT. Object detector 78 then provides the results of this detection as object data DO to map data generator 71.

[0181] Map data generator 71 is configured to generate map data MAP, including the occupied map, based on depth data DD, location movement data DPM, and object data DO.

[0182] The moving body information analysis unit 79 is configured to analyze the position and movement of objects detected by the object detector 78 relative to the UAV 3 based on depth data (DD), position movement data (DPM), and object data (DO). The moving body information analysis unit 79 then provides the results of this analysis as moving body data (DM) to the exposure frequency setting unit 76.

[0183] Similar to the exposure frequency setting unit 26 according to the first embodiment described above, the exposure frequency setting unit 76 is configured to set the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM. Furthermore, when an object is detected approaching the drone 3, the exposure frequency setting unit 76 also performs processing to correct the set exposure frequency based on the movement data DM provided from the movement information analysis unit 79, thereby increasing the exposure frequency of the stereo cameras 100 performing object imaging. Then, the exposure frequency setting unit 76 sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency.

[0184] Figure 20 An example of drone 3's operation is shown. In this example, bird 99 is flying around drone 3, and bird 99 is approaching drone 3. That is, the velocity vector V99 of bird 99 relative to drone 3 is pointing in the direction of drone 3.

[0185] The exposure frequency setting component 76 calculates the angle φ between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101 of each of the six stereo cameras 100. Specifically, with Figure 5 Similar to the case in [the previous example], the exposure frequency setting component 76 calculates the angle φA between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101A of the stereo camera 100A; calculates the angle φB between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101B of the stereo camera 100B; calculates the angle φC between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101C of the stereo camera 100C; calculates the angle φD between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101D of the stereo camera 100D; calculates the angle φE between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101E of the stereo camera 100E; and calculates the angle φF between the direction opposite to that indicated by the velocity vector V99 and the optical axis direction 101F of the stereo camera 100F. The angle φ is less than 180 degrees.

[0186] Then, the exposure frequency setting component 76 sets the exposure frequency to increase the exposure frequency of the stereo camera 100 corresponding to the smallest angle φ among the six angles φ. Specifically, for example, in Figure 14 In the example, the exposure frequency setting component 76 can set the exposure frequency F100 as follows, where F100A to F100D represent the exposure frequencies of stereo cameras 100A to 100D, respectively.

[0187] F100B>F100A>F100C>F100D

[0188] Here, the object detector 78 and the moving body information analysis component 79 correspond to a specific example of the "moving body detector" in this disclosure.

[0189] Figure 21 An operational example of controller 70 is shown. This flowchart is a flowchart of controller 20 according to the first embodiment described above, with steps S305 to S307 added. Figure 4 The flowchart is formed by this process.

[0190] Similar to the first embodiment described above, the position movement estimation unit 25 calculates the velocity vector VS of the UAV 3 based on image data DT, position data DP, and inertial data DI (step S101). Then, the exposure frequency setting unit 76 confirms whether the velocity value indicated by the velocity vector VS is equal to or less than the threshold TH1 (step S102).

[0191] In step S102, if the velocity value is not equal to or less than the threshold TH1 ("N" in step S102), the exposure frequency setting component 76 calculates the angle θ between the flight direction DIR of the UAV 3 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103). Then, the exposure frequency setting component 76 sets the respective exposure frequency of the six stereo cameras 100 based on the six angles θ (step S104).

[0192] Next, the moving body information analysis unit 79 analyzes the position and movement of the object detected by the object detector 78 relative to the UAV 3 based on the depth data DD, the position movement data DPM, and the object data DO (step S305).

[0193] Next, the exposure frequency setting unit 76 confirms the presence of an approaching object based on the analysis results of the moving object information analysis unit 79 (step S306). If no approaching object is found ("N" in step S306), the process proceeds to the processing in step S106.

[0194] In step S306, when an approaching object is present (Y in step S306), the exposure frequency setting unit 76 corrects the exposure frequency of each of the six stereo cameras 100 to increase the exposure frequency of the stereo camera 100 that performs imaging of the object (step S307). For example, the exposure frequency setting unit 76 calculates the angle φ between the direction opposite to the direction indicated by the velocity vector V99 and the optical axis direction 101 of each of the six stereo cameras 100, and sets the exposure frequency to increase the exposure frequency of the stereo camera 100 corresponding to the smallest angle φ among the six angles φ. Then, the exposure frequency setting unit 76 sets the exposure timing of each of the six stereo cameras 100 based on the set exposure frequency. Then, the process proceeds to the processing in step S106.

[0195] The following steps are similar to those in the first embodiment described above.

[0196] Therefore, in the UAV 3, the exposure frequencies of the six stereo cameras 100 are set based on the processing results of the object detector 78 and the moving object information analysis unit 79. Thus, in the UAV 3, for example, when an object is approaching, the exposure frequency of the stereo camera 100 that performs the imaging of that object can be increased. Therefore, in the UAV 3, the environment including the approaching object can be more accurately assessed, allowing for appropriate avoidance of the object. Therefore, for example, in the UAV 3, much of the computing resources of the controller 70 can be allocated to the computational processing of image data obtained by imaging the approaching object, enabling more accurate assessment of the environment while efficiently utilizing limited computing resources.

[0197] As described above, in this embodiment, the exposure frequencies of the six stereo cameras are set based on the processing results of the object detector and the moving body information analysis component, which allows for efficient use of computational resources. Other effects are similar to those in the first embodiment described above.

[0198] [Modified Example 3]

[0199] Any modifications to the first and second embodiments described above can be applied to the UAV 3 according to the above embodiments.

[0200] <4. Fourth Embodiment>

[0201] Next, the drone 4 according to the fourth embodiment will be described. In this embodiment, in addition to the six angles θ, the exposure frequency is set based on the overlap rate between image regions of images captured at different exposure times. It should be noted that components substantially the same as those of the drone 1 according to the first embodiment described above are indicated by the same reference numerals, and their descriptions are appropriately omitted.

[0202] Figure 22 An example configuration of drone 4 is shown. Drone 4 includes controller 80. Controller 80 includes exposure frequency setting component 86.

[0203] Similar to the exposure frequency setting unit 26 according to the first embodiment described above, the exposure frequency setting unit 86 is configured to set the exposure frequency of the six stereo cameras 100 in the imaging unit 11 based on the velocity vector VS included in the position movement data DPM. Furthermore, the exposure frequency setting unit 86 calculates the overlap rate between image regions in the captured images acquired at different exposure times in each of the six stereo cameras 100, based on the depth data DD and the position movement data DPM, and also performs processing for correcting the set exposure frequency based on the overlap rate. Then, the exposure frequency setting unit 86 sets the respective exposure time of the six stereo cameras 100 based on the set exposure frequency.

[0204] Figure 23 An operational example of UAV 4 is shown. In this example, a stereo camera 100F, which performs imaging in the downward direction of UAV 4, performs two exposure operations with different exposure timings. UAV 4 is moving, which causes the image region W1 in the first exposure operation EX and the image region W2 in the second exposure operation EX to be offset and not overlap. Therefore, as Figure 23 As shown, a portion of image region W1 and a portion of image region W2 overlap. Specifically, when the drone 4 moves at a high speed or the distance from the drone 4 to the imaging object is short, the overlapping image regions are widened. The exposure frequency setting unit 86 calculates an overlap rate, indicating the degree of overlap between image regions, based on the drone 4's moving speed and the distance from the drone 4 to the imaging object. Then, when the overlap rate between image regions in the captured image by the stereo camera 100F is low, the exposure frequency setting unit 86 increases the exposure frequency of the stereo camera 100F to improve the overlap rate.

[0205] Here, the depth data generator 24 corresponds to a specific example of the "distance calculation component" in this disclosure.

[0206] Figure 24 An operational example of controller 80 is shown. This flowchart is a flowchart of controller 20 according to the first embodiment described above, with steps S305 to S307 added. Figure 4 The flowchart is formed by this process.

[0207] Similar to the first embodiment described above, the position movement estimation unit 25 calculates the velocity vector VS of the UAV 4 based on image data DT, position data DP, and inertial data DI (step S101). Then, the exposure frequency setting unit 56 confirms whether the velocity value indicated by the velocity vector VS is equal to or less than the threshold TH1 (step S102).

[0208] In step S102, if the velocity value is not equal to or less than the threshold TH1 (N in step S102), the exposure frequency setting component 76 calculates the angle θ between the flight direction DIR of the UAV 4 indicated by the velocity vector VS and the optical axis direction 101 of each of the six stereo cameras 100 (step S103). Then, the exposure frequency setting component 76 sets the respective exposure frequency of the six stereo cameras 100 based on the six angles θ (step S104).

[0209] Next, the exposure frequency setting unit 86 calculates the overlap rate between image regions in the captured images obtained at different exposure times in each of the six stereo cameras 100, based on the moving speed of the drone 4 and the distance from the drone 4 to the imaging object (step S405). Specifically, the exposure frequency setting unit 86 calculates the overlap rate in stereo camera 100A; calculates the overlap rate in stereo camera 100B; calculates the overlap rate in stereo camera 100C; calculates the overlap rate in stereo camera 100D; calculates the overlap rate in stereo camera 100E; and calculates the overlap rate in stereo camera 100F.

[0210] Next, the exposure frequency setting unit 86 confirms whether at least one of the overlap rates of the six stereo cameras 100 is equal to or less than the threshold TH2 (step S406). If none of the overlap rates are equal to or less than the threshold TH2 ("N" in step S406), the process proceeds to the processing in step S106.

[0211] In step S406, if at least one of the overlap rates among the six stereo cameras 100 is equal to or less than a threshold TH2 (“Y” in step S406), the exposure frequency setting unit 86 corrects the respective exposure frequencies of the six stereo cameras 100 so that the overlap rates equal to or less than the threshold TH2 exceed the threshold TH2 (step S407). Then, the exposure frequency setting unit 86 sets the respective exposure timing of the six stereo cameras 100 based on the set exposure frequency. Then, the process proceeds to the processing in step S106.

[0212] The subsequent steps are similar to those in the first embodiment described above.

[0213] Therefore, in the drone 4, the exposure frequencies of the six stereo cameras 100 are set based on the drone 4's moving speed and the distance from the drone 4 to the imaging object. Thus, for example, in cases where the overlap between image regions is low, the exposure frequencies of the stereo cameras 100 with low overlap can be increased in the drone 4. Therefore, the overlap between image regions can be increased in the drone 4, which improves the accuracy of VIO processing in the position movement estimation unit 25, thereby improving the accuracy of the position and movement estimation of the drone 4.

[0214] As described above, in this embodiment, the exposure frequencies of the six stereo cameras are set according to the drone's moving speed and the distance from the drone to the imaging object, which improves the accuracy of the estimation of the drone's position and movement. Other effects are similar to those in the first embodiment described above.

[0215] [Modified Example 4]

[0216] Each of the modified examples of the first and second embodiments described above can be applied to the drone 4 according to the above embodiments.

[0217] Although the technology has been described above with reference to some embodiments and modified examples, the technology is not limited to these embodiments and can be modified in many ways.

[0218] For example, in the above embodiment, six stereo cameras 100 are provided, but this is not limiting. Instead, two or more, as well as five or fewer stereo cameras 100, may be provided, or seven or more stereo cameras 100 may be provided.

[0219] For example, in the above embodiment, a GPS receiver 12, a barometric pressure sensor 13, and an inertial measurement unit 14 are provided, but this is not limiting and some of them may be omitted.

[0220] Furthermore, while the above embodiments apply to drones, this technology is not limiting. Instead, it can be applied to robots, for example, that travel on planar surfaces.

[0221] It should be noted that the effects described in this article are illustrative and non-limiting only, and may include other effects.

[0222] It should be noted that this technology can have the following configurations. With this technology having the following configurations, computing resources can be used efficiently.

[0223] (1) An exposure control device, comprising:

[0224] The setting component calculates the camera angle between the movement direction of the device comprising multiple stereo cameras and the optical axis direction of each of the multiple stereo cameras, and sets the exposure frequency of each of the multiple stereo cameras based on the multiple camera angles; and

[0225] An exposure controller controls the operation of the plurality of stereo cameras based on multiple exposure frequencies set by a setting component.

[0226] (2) The exposure control device according to (1), wherein, when the moving speed of the device including the plurality of stereo cameras is faster than a predetermined speed, the setting component sets the exposure frequency of each of the plurality of stereo cameras based on the angle of the plurality of cameras.

[0227] (3) The exposure control device according to (1) or (2), wherein the setting component causes the exposure frequency of the first stereo camera with the smallest camera angle among the plurality of stereo cameras to be higher than the exposure frequency of the second stereo camera among the plurality of stereo cameras that is different from the first stereo camera.

[0228] (4) The exposure control device according to (3), wherein

[0229] When the moving speed of the device including the plurality of stereo cameras is a first speed, the setting component sets the exposure frequency of the first stereo camera to the first exposure frequency, and

[0230] When the moving speed of the device including the plurality of stereo cameras is a second speed that is faster than the first speed, the setting component sets the exposure frequency of the first stereo camera to a second exposure frequency that is higher than the first exposure frequency.

[0231] (5) The exposure control device according to any one of (1) to (4) further includes an environmental information generator, which generates environmental information about the surrounding environment of the device including the plurality of stereo cameras based on the imaging results of the plurality of stereo cameras, wherein

[0232] In addition to the multiple camera angles, the setting component also sets the exposure frequency of each of the multiple stereo cameras based on the environmental information.

[0233] (6) The exposure control device according to (5), wherein the setting component detects the width of the space in the surrounding environment based on the environmental information, and causes the exposure frequency of the third stereo camera among the plurality of stereo cameras that performs imaging in the direction of the width of the space to be higher than the exposure frequency of the fourth stereo camera among the plurality of stereo cameras that is different from the third stereo camera.

[0234] (7) The exposure control device according to (5) further includes an image analysis component that analyzes the imaging object based on the imaging results of the plurality of stereo cameras, wherein

[0235] The environmental information includes information about the object being imaged.

[0236] (8) The exposure control device according to any one of (1) to (4) further includes a moving body detector, which detects moving bodies around the device including the plurality of stereo cameras based on the imaging results of the plurality of stereo cameras, wherein

[0237] In addition to the multiple camera angles, the setting component also sets the exposure frequency of each of the multiple stereo cameras based on the detection results of the moving object detector.

[0238] (9) The exposure control device according to (8), wherein, when the distance between the moving body and the device including the plurality of stereo cameras narrows, a component is provided such that the exposure frequency of the fifth stereo camera among the plurality of stereo cameras that performs imaging of the moving body is higher than the exposure frequency of the sixth stereo camera among the plurality of stereo cameras that is different from the fifth stereo camera.

[0239] (10) The exposure control device according to any one of (1) to (4) further includes a distance calculation component for calculating the distance to the imaging object based on the imaging results of the plurality of stereo cameras, wherein

[0240] In addition to the multiple camera angles, the setting component also sets the exposure frequency of each of the multiple stereo cameras based on the moving speed of the device including the multiple stereo cameras and the distance to the imaging object.

[0241] (11) According to the exposure control device of (10), wherein the setting component calculates the overlap rate between image regions of two captured images with different exposure timings in one of the plurality of stereo cameras based on the moving speed of the device including the plurality of stereo cameras and the distance to the imaging object, and increases the exposure frequency of the stereo camera if the overlap rate is lower than a predetermined value.

[0242] (12) The exposure control device according to any one of (1) to (11), wherein

[0243] The setting component sets the exposure timing of each of the plurality of stereo cameras based on the exposure frequency of each of the plurality of stereo cameras, and

[0244] The exposure controller controls the operation of the multiple stereo cameras based on multiple exposure timings set by the setting component.

[0245] (13) The exposure control device according to (12), wherein the setting component sets the exposure timing of each of the plurality of stereo cameras so that the exposure timing of the plurality of stereo cameras is different from each other.

[0246] (14) The exposure control device according to (12) or (13) further includes an image acquisition component, which acquires the imaging results of the plurality of stereo cameras via a single bus wiring coupled to the plurality of stereo cameras, wherein

[0247] The image acquisition unit acquires the imaging results of the multiple stereo cameras at different times.

[0248] (15) The exposure control device according to (14), wherein

[0249] Each of the plurality of stereo cameras includes a plurality of image sensors.

[0250] The exposure timings of the multiple image sensors are the same for each other, and

[0251] The image acquisition unit acquires the imaging results of the multiple image sensors at different times.

[0252] (16) An exposure control method, comprising:

[0253] Calculate the camera angle between the movement direction of the device comprising multiple stereo cameras and the optical axis direction of each of the multiple stereo cameras, and set the exposure frequency of each of the multiple stereo cameras based on the multiple camera angles.

[0254] The operation of the multiple stereo cameras is controlled based on the set multiple exposure frequencies.

[0255] This application claims the benefit of Japanese priority patent application JP2020-011031, filed with the Japan Patent Office on January 27, 2020, the entire contents of which are incorporated herein by reference.

[0256] Those skilled in the art will understand that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors, as long as they are within the scope of the appended claims or their equivalents.

Claims

1. An exposure control device, comprising: The setting component calculates the camera angle between the movement direction of the drone, which includes multiple stereo cameras, and the optical axis direction of each of the multiple stereo cameras, and sets the exposure frequency of each of the multiple stereo cameras based on the multiple camera angles. and An exposure controller controls the operation of the plurality of stereo cameras based on multiple exposure frequencies set by a setting component. The setting component ensures that the exposure frequency of the first stereo camera, which has the smallest camera angle among the plurality of stereo cameras, is higher than the exposure frequency of the second stereo camera, which is different from the first stereo camera among the plurality of stereo cameras.

2. The exposure control apparatus according to claim 1, wherein When the drone moves faster than a predetermined speed, the setting component sets the exposure frequency of each of the plurality of stereo cameras based on the angles of the plurality of cameras.

3. The exposure control device according to claim 1, wherein... When the drone's moving speed is a first speed, the setting component sets the exposure frequency of the first stereo camera to the first exposure frequency, and When the drone's moving speed is a second speed that is faster than the first speed, the setting component sets the exposure frequency of the first stereo camera to a second exposure frequency that is higher than the first exposure frequency.

4. The exposure control device according to claim 1 further includes an environmental information generator, which generates environmental information about the surrounding environment of the UAV based on the imaging results of the plurality of stereo cameras, wherein... In addition to the multiple camera angles, the setting component also sets the exposure frequency of each of the multiple stereo cameras based on the environmental information.

5. The exposure control device according to claim 4, wherein the setting component detects the width of the space in the surrounding environment based on the environmental information, and causes the exposure frequency of the third stereo camera among the plurality of stereo cameras that performs imaging in the direction of the spatial width to be higher than the exposure frequency of the fourth stereo camera among the plurality of stereo cameras that is different from the third stereo camera.

6. The exposure control device according to claim 4 further includes an image analysis component that analyzes the imaging object based on the imaging results of the plurality of stereo cameras, wherein... The environmental information includes information about the object being imaged.

7. The exposure control device according to claim 1 further includes a moving object detector, which detects moving objects around the UAV based on the imaging results of the plurality of stereo cameras, wherein... In addition to the multiple camera angles, the setting component also sets the exposure frequency of each of the multiple stereo cameras based on the detection results of the moving object detector.

8. The exposure control device according to claim 7, wherein When the distance between the moving body and the drone narrows, the component is configured such that the exposure frequency of the fifth stereo camera, which performs imaging of the moving body, is higher than the exposure frequency of the sixth stereo camera, which is different from the fifth stereo camera.

9. The exposure control device according to claim 1, further comprising a distance calculation component for calculating the distance to the imaging object based on the imaging results of the plurality of stereo cameras, wherein... In addition to the multiple camera angles, the setting component also sets the exposure frequency of each of the multiple stereo cameras based on the drone's moving speed and its distance to the imaging object.

10. The exposure control device according to claim 9, wherein the setting component calculates the overlap rate between image regions of two captured images with different exposure timings in one of the plurality of stereo cameras based on the moving speed of the UAV and the distance to the imaging object, and increases the exposure frequency of the stereo camera if the overlap rate is lower than a predetermined value.

11. The exposure control device according to claim 1, wherein The setting component sets the exposure timing of each of the plurality of stereo cameras based on the exposure frequency of each of the plurality of stereo cameras, and The exposure controller controls the operation of the multiple stereo cameras based on multiple exposure timings set by the setting component.

12. The exposure control device of claim 11, wherein the setting component sets the exposure timing of each of the plurality of stereo cameras such that the exposure timings of the plurality of stereo cameras are different from each other.

13. The exposure control device according to claim 11, further comprising an image acquisition component, wherein the image acquisition component acquires the imaging results of the plurality of stereo cameras via a single bus wiring coupled to the plurality of stereo cameras, wherein... The image acquisition unit acquires the imaging results of the multiple stereo cameras at different times.

14. The exposure control device according to claim 13, wherein Each of the plurality of stereo cameras includes a plurality of image sensors. The exposure timings of the multiple image sensors are the same for each other, and The image acquisition unit acquires the imaging results of the multiple image sensors at different times.

15. An exposure control method, comprising: Calculate the camera angle between the movement direction of the drone, which includes multiple stereo cameras, and the optical axis direction of each of the multiple stereo cameras, and set the exposure frequency of each of the multiple stereo cameras based on the multiple camera angles. The operation of the multiple stereo cameras is controlled based on the set multiple exposure frequencies. The exposure frequency of each of the plurality of stereo cameras is set such that the exposure frequency of the first stereo camera with the smallest camera angle is higher than the exposure frequency of the second stereo camera, which is different from the first stereo camera.

Images