Information processing device and information processing method

Abstract

The present disclosure relates to an information processing device and an information processing method that allow imaging by an imaging device mounted on a flying body to be performed in a preferable manner. A light emission control unit switches control of a light-emitting device on the basis of the exposure timing of the imaging device mounted on the flying body and setting information of the light-emitting device, which is provided in the flying body or is present in a flight space of the flying body. The present disclosure can be applied to a drone equipped with an imaging device and various light-emitting devices.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method, and more particularly to an information processing device and an information processing method that enable suitable imaging by an imaging device mounted on an aircraft.

[0002] Conventionally, a technique for determining the self-position of a drone by using two-dimensional image data obtained by a camera mounted on the drone is known.

[0003] For example, Patent Literature 1 discloses a self-localization method in which a laser beam is projected onto a wall surface, image data including the laser beam projected onto the wall surface is acquired, and the posture of a moving object and the distance from the wall surface are determined based on the shape of the projected beam included in the image data. This method makes it possible to determine the position of a moving object with stable accuracy regardless of the shape of the structure.

[0004] International Publication No. 2021 / 006026

[0005] In the technique of Patent Document 1, it is necessary to capture the light emission in the image captured by the imaging device mounted on the aircraft, but there are cases where it is not desirable for the light emission to be captured in the captured image.

[0006] The present disclosure has been made in consideration of such circumstances, and aims to enable suitable realization of photography using an imaging device mounted on an aircraft.

[0007] The information processing device disclosed herein is an information processing device that includes a light-emitting control unit that switches control of a light-emitting device based on the exposure timing of a photographing device mounted on an aircraft and the setting information of a light-emitting device provided on the aircraft or present in the flight space of the aircraft.

[0008] The information processing method disclosed herein is an information processing method that includes switching control of a light-emitting device based on the exposure timing of an imaging device mounted on an aircraft and setting information of the light-emitting device provided on the aircraft or present in the flight space of the aircraft.

[0009] In the present disclosure, control of the light-emitting device is switched based on the exposure timing of the imaging device mounted on the aircraft and the setting information of the light-emitting device installed on the aircraft or present in the flight space of the aircraft.

[0010] 1 is a perspective view showing the appearance of a drone. FIG. 1 is a diagram illustrating a first example of light emission control of the technology according to the present disclosure. FIG. 2 is a diagram illustrating a second example of light emission control of the technology according to the present disclosure. FIG. 3 is a diagram illustrating a third example of light emission control of the technology according to the present disclosure. A block diagram showing an example functional configuration of an information processing unit. FIG. 1 is a block diagram showing an example configuration of a first embodiment of a drone. FIG. 2 is a flowchart showing the flow of photography processing by a drone. FIG. 2 is a block diagram showing an example configuration of a second embodiment of a drone. FIG. 3 is a flowchart showing the flow of photography processing by a drone. FIG. 3 is a block diagram showing an example configuration of a third embodiment of a drone. FIG. 4 is a flowchart showing the flow of photography processing by a drone. FIG. 4 is a diagram showing an example of light emission control based on a priority camera. FIG. 5 is a diagram showing reflections depending on the viewing angle of the camera. FIG. 5 is a diagram showing reflections depending on the sensitivity band of the camera. A flowchart showing interference detection processing. FIG. 6 is a diagram showing an example application of light emission control of the technology according to the present disclosure. FIG. 7 is a diagram showing an example application of light emission control of the technology according to the present disclosure. FIG. 8 is a diagram showing an example application of light emission control of the technology according to the present disclosure. FIG. 9 is a block diagram showing an example configuration of computer hardware.

[0011] Modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described below in the following order.

[0012] 1. Drone appearance 2. Overview of light emission control of the technology according to the present disclosure 3. Functional configuration of an information processing unit to which the technology according to the present disclosure is applied 4. First embodiment (integrated control of photography and light emission) 5. Second embodiment (integrated control of light emission) 6. Third embodiment (individual control of light emission) 7. Light emission control based on a priority camera 8. Light emission interference detection 9. Application example 10. Example of computer hardware configuration

[0013] 1. Appearance of Drone FIG. 1 is a perspective view showing the appearance of a drone, which is an air vehicle to which the technology according to the present disclosure can be applied.

[0014] The drone 1 shown in Figure 1 can move in any direction by remote control flight or autonomous flight.

[0015] Although detailed description of each part of the drone 1 will be omitted, a gimbal 10 is detachably attached to the bottom of the main body of the drone 1. An RGB camera 11 is detachably attached to the tip of the gimbal 10 as a photographing device mounted on an aircraft in the technology disclosed herein.

[0016] A stereo camera 12 is provided as an imaging device mounted on an aircraft according to the technology disclosed herein on the side of the drone 1 body. Stereo camera images obtained by the stereo camera 12 are used for self-position estimation processing of the drone 1.

[0017] Multiple frame parts (four in this embodiment) extend from the main body of the drone 1. Of these, two frame parts extending to the left and right in the forward direction of the drone 1 (the shooting direction of the RGB camera 11) in the figure are each provided with direction identification lights 13L, 13R that indicate the traveling direction of the drone 1. The light emission (illumination) of the direction identification lights 13L, 13R allows the operator (user) operating the drone 1 to understand the traveling direction of the drone 1.

[0018] The gimbal 10 is also equipped with a projector 14 (not shown) that includes a projection unit capable of projecting a predetermined pattern, along with the RGB camera 11. The pattern projected by the projection unit is used to grasp the shape of a target object, such as for inspecting a structure.

[0019] Although not shown, various electrical component units and the like are mounted inside the main body and frame of the drone 1. Specifically, a receiving unit, a controller, a sensor unit, a battery unit, a cooling fan, and the like are mounted inside the main body of the drone 1. A drive control unit that controls the drive of the motor that rotates the propellers and the like are mounted inside the frame.

[0020] 2. Overview of Light Emission Control of the Technology According to the Present Disclosure In the technology according to the present disclosure, the light emission of the light emitting device provided on the drone 1 and the light emitting device present in the flight space of the drone 1 is controlled in conjunction with the exposure timing of the imaging device mounted on the drone 1 described with reference to FIG.

[0021] FIG. 2 is a diagram illustrating a first example of light emission control according to the technology of the present disclosure.

[0022] 2, in conjunction with the exposure timing (shutter timing) of the RGB camera 11 mounted on the drone 1, the stereo camera 12 equipped on the drone 1 takes an image and the light emission of the direction identification light 13 provided on the drone 1 is turned off. In other words, neither the RGB camera 11 nor the stereo camera 12 captures the light emission of the direction identification light 13.

[0023] This makes it possible to prevent unwanted reflections in the image captured by the RGB camera 11 and the stereo camera image captured by the stereo camera 12.

[0024] The image captured by the RGB camera 11 may be a still image captured using a shutter operation by the user as a capture trigger, or a moving image captured at a predetermined shutter speed (frame rate).

[0025] When capturing moving images using the RGB camera 11, the shutter period is so short that the period during which the direction identification light 13 is off is also so short that it cannot be visually detected by the human eye. Therefore, even if the direction identification light 13 is turned off in conjunction with the shutter speed (frame rate) of the RGB camera 11, it will not affect visibility to the pilot, or will only cause a slight decrease in brightness to be noticed.

[0026] FIG. 3 is a diagram illustrating a second example of light emission control according to the technology of the present disclosure.

[0027] 3, in conjunction with the exposure timing (shutter timing) of the RGB camera 11 mounted on the drone 1, the stereo camera 12 equipped on the drone 1 is turned off and the light emission (pattern projection) of the projector 14 provided on the drone 1 is turned on. In other words, the RGB camera 11 captures the projection pattern of the projector 14, but the stereo camera 12 does not capture the projection pattern of the projector 14.

[0028] This makes it possible to highlight features such as unevenness of the projection target when, for example, a predetermined pattern projected by projector 14 onto a projection target such as a wall surface to be inspected is captured by RGB camera 11. In this way, the image including the pattern captured by RGB camera 11 can be used for analytical processing such as grasping the shape of the projection target onto which the pattern is projected.

[0029] FIG. 4 is a diagram illustrating a third example of light emission control according to the technology of the present disclosure.

[0030] 4 , in conjunction with the exposure timing (shutter timing) of the RGB camera 11 mounted on the drone 1, the stereo camera 12 provided on the drone 1 is turned off to capture images, and the light emission (pattern projection) of the projector 14 provided on the drone 1 is also turned off. In other words, the light emission (pattern projection) of the projector 14 is turned on in conjunction with the exposure timing (shutter timing) of the stereo camera 12. In other words, the stereo camera 12 captures the projection pattern of the projector 14, but the RGB camera 11 does not capture the projection pattern of the projector 14.

[0031] As a result, a stereo camera image including feature points can be obtained by capturing a predetermined (random) pattern projected by the projector 14 in an environment with low texture, a continuous regular pattern, or dim lighting, for example. In this way, the image including the pattern captured by the stereo camera 12 can be used for the self-position estimation process of the drone 1.

[0032] A specific configuration for realizing the above-described light emission control will be described below.

[0033] 3. Functional Configuration of an Information Processing Unit to which the Technology According to the Present Disclosure is Applied FIG. 5 is a block diagram showing an example of the functional configuration of an information processing unit to which the technology according to the present disclosure is applied.

[0034] 5 is realized by, for example, a processor mounted on the drone 1. The drone 1 is also equipped with imaging devices 110-1 and 110-2 and light emitting devices 120-1, 120-2, . . .

[0035] The image capturing devices 110-1 and 110-2 (hereinafter simply referred to as image capturing devices 110) correspond to the RGB camera 11 and the stereo camera 12 described with reference to Fig. 1. However, the image capturing devices 110 may also be equipped with an infrared camera, a depth camera, or the like.

[0036] The light-emitting devices 120-1, 120-2, ... (hereinafter simply referred to as light-emitting devices 120) correspond to the direction identification light 13 and projector 14 described with reference to Figure 1. Without being limited to this, the drone 1 may be equipped with an infrared ranging sensor such as a 1d ToF (Time of Flight) sensor or LiDAR (Light Detection and Ranging), a beacon, a diffractive optical element that emits laser pattern light, a camera flash, or the like as the light-emitting device 120. Furthermore, the light-emitting device 120 may include an external light-emitting device that is provided outside the drone 1 and exists in the flight space of the drone 1, such as an emergency light, spatial lighting equipment, a laser pointer, or a ground-mounted projector.

[0037] Setting information for the image capturing device 110 and setting information for the light emitting device 120 are input to the information processing unit 100. The setting information for the image capturing device 110 includes the shutter speed (frame rate), viewing angle, sensitivity band, etc. of the image capturing device 110. The setting information for the light emitting device 120 includes the light emission on / off cycle of the light emitting device 120, the light emission wavelength band, the positional relationship with the image capturing device 110, the light emission range, etc.

[0038] The information processing unit 100 executes a program stored in a memory not shown, etc., to realize functional blocks including a photography control unit 131, a light emission control unit 132, an arithmetic processing unit 133, a flight control unit 134, and an interference determination unit 135.

[0039] The imaging control unit 131 controls the imaging of each imaging device 110 and acquires various images (data) captured by the imaging devices 110 .

[0040] The light emission control unit 132 switches the control of each light emitting device 120 based on the exposure timing of the image capturing device 110 and the setting information of each light emitting device 120 .

[0041] For example, the light emission control unit 132 switches on / off the light emission of the light-emitting devices 120 with an emission wavelength band that can be captured by the image capturing device 110, according to the exposure timing of the image capturing device 110. Furthermore, the light emission control unit 132 does not control the light emission of the light-emitting devices 120 with an emission wavelength band that cannot be captured by the image capturing device 110. Furthermore, the light emission control unit 132 does not control the light emission of the light-emitting devices 120 with a positional relationship or emission range that does not include the light emission within the image capturing range of the image capturing device 110. The light emission control unit 132 can also determine whether or not to control the light emission of each light-emitting device 120.

[0042] The arithmetic processing unit 133 executes predetermined arithmetic processing based on the image captured by the image capturing device 110 .

[0043] Specifically, the arithmetic processing unit 133 performs preprocessing on an image including a pattern projected by the projector 14 or the like in order to analyze the projection target onto which the pattern is projected. Furthermore, the arithmetic processing unit 133 executes self-position estimation processing of the drone 1 based on the image including the pattern projected by the projector 14 or the like.

[0044] The flight control unit 134 controls the flight of the drone 1 by controlling a drive control unit (not shown) based on the results of the self-position estimation process executed by the calculation processing unit 133.

[0045] The interference determination unit 135 determines whether the light emitted by the light emitting device 120 is included in the shooting range of the shooting device 110, and generates an interference signal indicating that the light emitted by the light emitting device 120 is interfering with the shooting by the shooting device 110.

[0046] Specifically, the interference determination unit 135 determines whether the light emitted by the light-emitting device 120 is included in the imaging range of the imaging device 110, based on the viewing angle of the imaging device 110 and the light emission range of the light-emitting device 120, or based on the sensitivity band of the imaging device 110 and the emission wavelength band of the light-emitting device 120. The interference determination unit 135 adds an interference signal to images captured by the imaging device 110 during a period in which the light emitted by the light-emitting device 120 is included in the imaging range of the imaging device 110. In this case, the calculation processing unit 133 performs calculation processing, excluding images to which the interference signal has been added.

[0047] According to the above configuration, the control of each light-emitting device 120 is switched based on the exposure timing of the imaging device 110 and the setting information of each light-emitting device 120. This makes it possible to preferably realize imaging by the imaging device 110 mounted on the drone 1, such as by making the light emission appear in the image captured by the imaging device 110, or conversely, by making it possible to prevent the light emission from appearing in the captured image.

[0048] Below, we will explain an embodiment of a drone 1 to which the technology disclosed herein is applied.

[0049] 4. First Embodiment (Integrated Control of Photography and Light Emission) FIG. 6 is a block diagram showing an example configuration of a first embodiment of a drone 1 to which the technology according to the present disclosure is applied.

[0050] 6 includes an application processor 211, a vision sensing system 212, a flight controller 213, a communication module 214, and an integrated controller 215. The drone 1 is also equipped with an RGB camera 11 and a stereo camera 12 as imaging devices (hereinafter simply referred to as cameras), a direction identification light 13 and a projector 14 as light-emitting devices, and an infrared ranging sensor 221 configured with a 1d ToF sensor, LiDAR, or the like.

[0051] The application processor 211 mainly controls image capture by the RGB camera 11 and wireless communication by the communication module 214. The vision sensing system 212 mainly executes self-position estimation processing based on stereo camera images from the stereo camera 12. The flight controller 213 is configured as a drive control unit, and mainly executes flight control of the drone 1 based on sensor data from sensors (not shown). The communication module 214 performs wireless communication with the transmitter 230 operated by the pilot.

[0052] The transmitter 230 includes an application processor 241 that realizes each function, a communication module 242 that performs wireless communication with the drone 1, and an output unit 243 that outputs various information to external devices. The output unit 243 is connected wirelessly or by wire to an external light-emitting device 250 configured as an emergency light, spatial lighting equipment, a laser pointer, a ground-mounted projector, or the like.

[0053] The integrated controller 215 has the functions of the imaging control unit 131 and the light emission control unit 132 of the information processing unit 100 described with reference to Fig. 5. The integrated controller 215 controls imaging by the RGB camera 11 and the stereo camera 12 as cameras by generating an imaging trigger. The integrated controller 215 also controls the light emission of the direction identification light 13, the projector 14, and the infrared distance measuring sensor 221 as light-emitting devices by generating control signals for controlling the light emission (on / off, measurement / off) of these light-emitting devices. Furthermore, the integrated controller 215 also controls the light emission (on / off) of the external light-emitting device 250 by wireless communication with the transmitter 230 via the communication module 214.

[0054] The flow of the photographing process by the drone 1 in Fig. 6 will be described with reference to the flowchart in Fig. 7. In the process in Fig. 7, photographing is performed by the RGB camera 11, but photographing may also be performed by other cameras mounted on the drone 1, such as the stereo camera 12.

[0055] In step S11, the integrated controller 215 determines whether or not to control the light emission of each light-emitting device (the direction identification light 13, the projector 14, the infrared distance measuring sensor 221, and the external light-emitting device 250). If it is determined that the light emission of each light-emitting device is to be controlled, the process proceeds to step S12.

[0056] In step S12, the integrated controller 215 calculates the on / off timing of each light-emitting device based on the exposure timing of each camera. For example, when the RGB camera 11 captures moving images, the exposure timing of the RGB camera 11 can be determined based on the shutter speed of the RGB camera 11. When the RGB camera 11 captures still images, the exposure timing of the RGB camera 11 can be determined based on the flight plan of the drone 1.

[0057] In step S13, the integrated controller 215 starts controlling the on / off of each light emitting device based on the calculated on / off timing of each light emitting device.

[0058] Then, in step S14, the integrated controller 215 controls each camera while controlling the on / off of each light emitting device, thereby performing photography based on the flight plan.

[0059] On the other hand, if it is determined in step S11 that the light emission of each light emitting device is not to be controlled, steps S12 and S13 are skipped, and photography is performed by each camera in step S14.

[0060] In step S15, the integrated controller 215 determines whether or not the photographing based on the flight plan has been completed. If the photographing based on the flight plan has not yet been completed, the process returns to step S14, and the photographing based on the flight plan is repeated. On the other hand, if the photographing based on the flight plan has been completed, the photographing process ends.

[0061] 5. Second embodiment (integrated control of light emission) FIG. 8 is a block diagram showing a configuration example of a second embodiment of the drone 1 to which the technology according to the present disclosure is applied.

[0062] In the drone 1 shown in Fig. 8, the same components as those in the drone 1 shown in Fig. 6 are denoted by the same reference numerals, and the description thereof will be omitted. Also, in Fig. 8, the transmitter 230 and the external light-emitting device 250 are omitted from illustration.

[0063] In the drone 1 shown in Fig. 8, the integrated controller 215 has the function of the light emission control unit 132 of the information processing unit 100 described with reference to Fig. 5. The integrated controller 215 does not control the imaging of the RGB camera 11 and the stereo camera 12 as cameras, but controls the light emission of these light-emitting devices by generating control signals for controlling the light emission (on / off, measurement / off) of the direction identification light 13, the projector 14, and the infrared distance measuring sensor 221 as light-emitting devices based on imaging triggers from each imaging device.

[0064] The flow of the photographing process by the drone 1 of Fig. 8 will be described with reference to the flowchart of Fig. 9. In the process of Fig. 9, photographing is performed by the RGB camera 11, but photographing may also be performed by other cameras mounted on the drone 1, such as the stereo camera 12.

[0065] In step S21, the integrated controller 215 determines whether or not to control the light emission of each light-emitting device (the direction identification light 13, the projector 14, the infrared distance measuring sensor 221, and the external light-emitting device 250) provided on the drone 1. If it is determined that the light emission of each light-emitting device is to be controlled, the process proceeds to step S22.

[0066] In step S22, the integrated controller 215 acquires the exposure timing of each camera. For example, when the RGB camera 11 captures a moving image, the exposure timing of the RGB camera 11 can be acquired based on the shutter speed of the RGB camera 11. For example, when the RGB camera 11 captures a still image, the exposure timing of the RGB camera 11 can be acquired based on the flight plan of the drone 1.

[0067] In step S23, the integrated controller 215 calculates the on / off timing of each light emitting device based on the acquired exposure timing of each camera.

[0068] In step S24, the integrated controller 215 starts controlling the on / off of each light emitting device based on the calculated on / off timing of each light emitting device.

[0069] Then, in step S25, the integrated controller 215 controls each camera while controlling the on / off of each light emitting device, thereby performing photography based on the flight plan.

[0070] On the other hand, if it is determined in step S21 that the light emission of each light emitting device is not to be controlled, steps S22 to S24 are skipped, and photography is performed by each camera in step S25.

[0071] In step S26, the integrated controller 215 determines whether or not the photographing based on the flight plan has been completed. If the photographing based on the flight plan has not yet been completed, the process returns to step S25, and the photographing based on the flight plan is repeated. On the other hand, if the photographing based on the flight plan has been completed, the photographing process ends.

[0072] 6. Third Embodiment (Individual Control of Light Emission) FIG. 10 is a block diagram showing a configuration example of a third embodiment of the drone 1 to which the technology according to the present disclosure is applied.

[0073] In the drone 1 shown in Fig. 10, the same components as those in the drone 1 shown in Fig. 6 are denoted by the same reference numerals, and the description thereof will be omitted. Also, in Fig. 10, the transmitter 230 and the external light-emitting device 250 are omitted from illustration.

[0074] 10 does not include an integrated controller 215, and an RGB camera 11 serving as a camera has the function of the light emission control unit 132 of the information processing unit 100 described with reference to Fig. 5. The RGB camera 11 outputs a shooting trigger input by a user as a control signal for controlling the light emission (on / off, measurement / off) of the direction identification light 13, projector 14, and infrared distance measuring sensor 221 serving as light-emitting devices, thereby controlling the light emission of these light-emitting devices.

[0075] The flow of the photographing process by the drone 1 of Fig. 10 will be described with reference to the flowchart of Fig. 11. In the process of Fig. 11, the RGB camera 11 captures a still image using a user's shutter operation as a photographing trigger. That is, the exposure timing of the RGB camera 11 is determined based on the photographing trigger input to the RGB camera 11. The process of Fig. 11 may be executed each time a photographing trigger is input to the RGB camera 11.

[0076] In step S31, the RGB camera 11 determines whether to control the light emission of each light-emitting device (the direction identification light 13, the projector 14, the infrared distance measuring sensor 221, and the external light-emitting device 250) provided on the drone 1. If it is determined that the light emission of each light-emitting device is to be controlled, the process proceeds to step S32.

[0077] In step S32, the RGB camera 11 outputs a photographing trigger input in response to the user's shutter operation to each light emitting device.

[0078] In step S33, the RGB camera 11 controls the on / off of each light emitting device based on the exposure timing of the RGB camera 11, using the photographing trigger output to each light emitting device as a control signal.

[0079] Then, in step S34, the RGB camera 11 performs photography in response to the input photography trigger.

[0080] On the other hand, if it is determined in step S31 that the light emission of each light emitting device is not to be controlled, steps S32 and S33 are skipped, and photography is performed by the RGB camera 11 in step S34.

[0081] The above-described process is repeated every time a photographing trigger is input to the RGB camera 11 .

[0082] 7. Light Emission Control Based on Priority Camera In the light emission control of the technology disclosed herein, the control of the light emitting devices can be switched based on the exposure timing of a designated priority camera (priority photographing device) among the multiple cameras mounted on the drone 1 and the setting information of the multiple light emitting devices. For example, the light emission of a desired light emitting device among the multiple light emitting devices can be switched on / off so that only the light emitted by the desired light emitting device is photographed by the priority camera.

[0083] FIG. 12 is a diagram illustrating an example of light emission control based on a priority camera.

[0084] Figure 12 shows a timing chart illustrating the operation of the RGB camera 11 and stereo camera 12 as cameras mounted on the drone 1, the direction identification light 13 as a light-emitting device, the infrared ranging sensor 221, and the projector 14.

[0085] 12, the RGB camera 11 designated as the priority camera captures a still image including a pattern projected by the projector 14 serving as the desired light-emitting device in response to a user's shutter operation. The exposure timing (photographing cycle) of the stereo camera 12 and the light-emitting cycles of the direction identification light 13 and the infrared distance measuring sensor 221 are all different.

[0086] First, the light emission (projection of the pattern) of the projector 14 is controlled so that the pattern projected by the projector 14 is photographed by the RGB camera 11, which is the priority camera, and the light is turned on / off in accordance with the photographing timing of the RGB camera 11.

[0087] Next, in order to prevent the light emitted by the direction identification light 13 from being captured by the RGB camera 11, which is the priority camera, the light emission of the direction identification light 13 is controlled so that it is turned off at a time when there is a possibility that it will be captured by the RGB camera 11, even if the light emission is on.

[0088] Note that, due to the relationship between the viewing angle of the RGB camera 11 and the light emission range (light emission irradiation range) of the direction identification light 13, the light emission of the direction identification light 13 is not controlled if the light emission of the direction identification light 13 is not captured by the RGB camera 11. For example, as shown in Fig. 13, if the viewing range 311 of the RGB camera 11 and the light emission irradiation range 313R of the direction identification light 13R that illuminates the front right of the drone 1 do not overlap, but the light emission irradiation range 313L of the direction identification light 13L that illuminates the front left of the drone 1 overlap, the light emission of the direction identification light 13L is controlled so that the light emission is turned off in accordance with the shooting timing of the RGB camera 11.

[0089] Furthermore, if the light emitted by the infrared ranging sensor 221 is within the field of view of the RGB camera 11 but is not captured, the light emission of the infrared ranging sensor 221 is not controlled. Specifically, as shown in Fig. 14 , the sensitivity band of the RGB camera 11, which is the priority camera, does not include the emission wavelength band of the infrared ranging sensor 221, and the light emitted by the infrared ranging sensor 221 is not captured by the RGB camera 11, so the light emission of the infrared ranging sensor 221 does not need to be controlled. Note that if a near-infrared camera is mounted as the camera of the drone 1, the sensitivity band of the near-infrared camera includes the emission wavelength band of the infrared ranging sensor 221, and therefore the light emission of the infrared ranging sensor 221 needs to be controlled so that the light emission is turned off in accordance with the timing of capturing images by the near-infrared camera.

[0090] On the other hand, the stereo camera 12 that is not the priority camera captures unwanted light emission, such as light emitted by the projector 14 that is turned on in synchronization with the image capture timing of the RGB camera 11.

[0091] <8. Light Emission Interference Determination> In cases where light emission that is not actually desired to be captured is captured, the interference determination unit 135 of the information processing unit 100 described with reference to FIG. 5 generates an interference signal indicating that the light emission of the light emitting device is interfering with the image capture by the camera.

[0092] Here, the light emission interference detection process performed by the interference detection unit 135 will be described with reference to the flowchart in Fig. 15. The process in Fig. 15 may be executed while the RGB camera 11 and the stereo camera 12 mounted on the drone 1 are capturing images.

[0093] In step S111, the interference determination unit 135 determines whether or not there is light emission from each light-emitting device (the direction identification light 13, the projector 14, the infrared distance measuring sensor 221, and the external light-emitting device 250) provided on the drone 1. If it is determined that there is light emission from at least one of the light-emitting devices, the process proceeds to step S112.

[0094] In step S112, the interference determination unit 135 determines whether the light emission from the light-emitting devices is included in the camera's shooting range by determining whether the viewing angle (camera field of view) of the RGB camera 11 or the stereo camera 12 overlaps with the illumination range of each light-emitting device. If it is determined that the camera field of view does not overlap with the illumination range of each light-emitting device, the process proceeds to step S113.

[0095] In step S113, the interference determination unit 135 determines whether the light emission from the light-emitting devices is included in the camera's imaging range by determining whether the sensitivity bands (camera sensitivity bands) of the RGB camera 11 and the stereo camera 12 do not overlap with the emission wavelength bands of the light-emitting devices. If it is determined that the camera sensitivity bands do not overlap with the emission wavelength bands of the light-emitting devices, it is determined that the light emission from the light-emitting devices does not interfere with imaging by the cameras, and the interference determination process ends.

[0096] If it is determined in step S111 that no light is emitted from the light-emitting devices, steps S112 and S113 are skipped and the collision detection process ends.

[0097] On the other hand, if it is determined in step S112 that the camera field of view and the illumination range of each light-emitting device overlap, or if it is determined in step S113 that the camera sensitivity band and the emission wavelength band of each light-emitting device overlap, proceed to step S114.

[0098] In step S114, the interference determination unit 135 generates an interference signal indicating that the light emitted from the light emitting device is interfering with the image capturing by the camera.

[0099] In step S115, the interference determination unit 135 assigns the generated interference signal to an image captured by the camera during a period in which the light emitted by the light emitting device is included in the imaging range of the camera.

[0100] The image captured by the camera is supplied to a calculation processing unit 133 that executes predetermined calculation processing. At this time, the interference signal may be supplied to the calculation processing unit 133 separately from the image, or the image to which the interference signal is added as metadata may be supplied to the calculation processing unit 133. Furthermore, the interference signal may be saved in a file separate from the image.

[0101] For example, when the arithmetic processing unit 133 performs a self-location estimation process for the drone 1 based on stereo camera images captured by the stereo camera 12, if unnecessary light emission is captured in the stereo camera images, the accuracy of the self-location estimation process will deteriorate. Therefore, the arithmetic processing unit 133 can prevent the accuracy of the self-location estimation process from deteriorating even if unnecessary light emission is captured in the stereo camera images by excluding stereo camera images to which an interference signal has been added. In this case, if the interference signal is stored in a file separate from the images, the stereo camera images containing unnecessary light emission may be excluded by referencing that file.

[0102] 9. Application Examples Hereinafter, application examples of light emission control of the technology according to the present disclosure will be described.

[0103] (First Application Example) FIG. 16 is a diagram illustrating a first application example of light emission control of the technology according to the present disclosure.

[0104] 2, in the example of Fig. 16, the stereo camera 12 takes an image and turns off the light emission (pattern projection) of the projector 14 in conjunction with the exposure timing (shutter timing) of the RGB camera 11. As a result, the projection pattern of the projector 14 is not captured by either the RGB camera 11 or the stereo camera 12.

[0105] When this type of light emission control is applied to projecting patterns to alert third parties TP, such as projecting an arrow P11 indicating the direction of travel of the drone 1, these patterns are not captured by either the RGB camera 11 or the stereo camera 12, so any pattern can be projected.

[0106] Second Application Example FIG. 17 is a diagram illustrating a second application example of light emission control of the technology according to the present disclosure.

[0107] 17, as described with reference to Fig. 3, the stereo camera 12 is turned off and the projector 14 is turned on to emit light (project a pattern) in conjunction with the exposure timing (shutter timing) of the RGB camera 11. As a result, the RGB camera 11 captures the projection pattern of the projector 14, but the stereo camera 12 does not capture the projection pattern of the projector 14.

[0108] When such light emission control is applied to the projection of patterns P21 for analysis processing of a projection target such as a wall surface to be inspected, these patterns P21 are not captured by the stereo camera 12, so an image of the projection target onto which the pattern P21 to be analyzed is projected can be captured without affecting the self-position estimation processing using the stereo camera image.

[0109] As shown in Fig. 17, by changing the pattern P21 over time, it is possible to change the features that stand out on the projection target. Alternatively, as shown in Fig. 18, the projection of the pattern P22 may be switched on and off over time. In this case, the features of the projection target can be extracted from the difference between the image captured when the projection is on and the image captured when the projection is off. It is also possible to extract the features of the projection target using only the images captured when the projection is on, and also to extract the features of the projection target using only the images captured when the projection is off.

[0110] (Third Application Example) FIG. 19 is a diagram illustrating a third application example of light emission control of the technology according to the present disclosure.

[0111] 19, as described with reference to Fig. 4, the stereo camera 12 is turned off in conjunction with the exposure timing (shutter timing) of the RGB camera 11, and the light emission (pattern projection) of the external light-emitting device 250, which is a ground-mounted projector, is also turned off. As a result, the stereo camera 12 captures the projection pattern of the external light-emitting device 250, but the RGB camera 11 does not capture the projection pattern of the external light-emitting device 250.

[0112] When such light emission control is applied to the projection of a pattern P31 containing feature points for the self-position estimation process of the drone 1 in an environment such as one with low texture, continuous regular patterns, or dim lighting, the pattern P31 is not captured by the RGB camera 11, so a stereo camera image containing feature points used for the self-position estimation process can be obtained without affecting the photography by the RGB camera 11.

[0113] (Other Modifications) In the above-described embodiment, the image capturing device mounted on the drone 1 is configured as an RGB camera, a stereo camera, an infrared camera, a depth camera, etc. However, the present disclosure is not limited to this and can be applied to an air vehicle equipped with a light detection device capable of outputting an electrical signal in response to received light, instead of the image capturing device.

[0114] 10. Example of Computer Hardware Configuration The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, a general-purpose personal computer, or the like.

[0115] 20 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. The information processing unit 100 that may be included in the drone 1 may be configured, for example, by a computer 500 having a configuration similar to that shown in FIG.

[0116] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0117] An input / output interface 505 is also connected to the bus 504. An input unit 506 including buttons, a touch panel, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0118] In the computer 500 configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the memory unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0119] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0120] The program executed by computer 500 may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0121] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0122] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0123] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0124] For example, the embodiment of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0125] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0126] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0127] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0128] Furthermore, the technology according to the present disclosure may have the following configurations. (1) An information processing device including a light emission control unit that switches control of a light emitting device based on the exposure timing of an image capturing device mounted on an air vehicle and setting information of a light emitting device provided on the air vehicle or present in the flight space of the air vehicle. (2) The information processing device described in (1), in which the setting information includes at least one of an emission on / off cycle of the light emitting device, an emission wavelength band, a positional relationship with the image capturing device, and an emission range. (3) The information processing device described in (2), in which the light emission control unit switches on / off emission of the light emitting device in the emission wavelength band that the image capturing device can capture, based on the exposure timing. (4) The information processing device described in (3), in which the light emission control unit does not control emission of the light emitting device in the emission wavelength band that the image capturing device cannot capture. (5) The information processing device described in (2), in which the light emission control unit does not control emission of the light emitting device in the positional relationship or emission range where the emission is not included in the image capturing range of the image capturing device. (6) The information processing device according to any one of (1) to (5), further comprising an interference determination unit that determines whether or not light emission from the light-emitting device is included in the imaging range of the imaging device, and generates an interference signal indicating that light emission from the light-emitting device is interfering with imaging by the imaging device. (7) The information processing device according to (6), wherein the interference determination unit determines whether or not light emission from the light-emitting device is included in the imaging range based on the viewing angle of the imaging device and the light-emitting range of the light-emitting device, or based on the sensitivity band of the imaging device and the emission wavelength band of the light-emitting device. (8) The information processing device according to (6), wherein the interference determination unit assigns the interference signal to images captured by the imaging device during a period when light emission from the light-emitting device is included in the imaging range. (9) The information processing device according to (8), further comprising an arithmetic processing unit that performs predetermined arithmetic processing based on the images captured by the imaging device, wherein the arithmetic processing unit performs the arithmetic processing by excluding the images to which the interference signal is assigned. (10) The information processing device according to (1), wherein the light-emitting device has a projection unit that can project a predetermined pattern.(11) The information processing device according to (10), wherein an image including the pattern captured by the imaging device is used for analysis processing of a projection target object onto which the pattern is projected. (12) The information processing device according to (10), further comprising an arithmetic processing unit that executes predetermined arithmetic processing based on the image including the pattern captured by the imaging device. (13) The information processing device according to (12), wherein the arithmetic processing includes self-position estimation processing of the flying object. (14) The information processing device according to (1), wherein the exposure timing is determined based on a shutter speed of the imaging device. (15) The information processing device according to any of (1) to (14), wherein the exposure timing is determined based on a flight plan of the flying object. (16) The information processing device according to any of (1) to (14), wherein the exposure timing is determined based on an imaging trigger input to the imaging device. (17) The information processing device according to any of (1) to (14), wherein the light emission control unit switches control of the light emitting devices based on the exposure timing of a designated priority imaging device among the plurality of imaging devices mounted on the flying object and the setting information of the plurality of light emitting devices. (18) The information processing device according to (17), wherein the light emission control unit switches on / off light emission of a desired light-emitting device among the plurality of light-emitting devices so that only light emission of the desired light-emitting device is captured by the priority photographing device. (19) The information processing device according to any of (1) to (18), wherein the photographing device is configured with any of an RGB camera, a stereo camera, an infrared camera, and a depth camera. (20) An information processing method including: switching control of the light-emitting device based on exposure timing of a photographing device mounted on an aircraft and setting information of a light-emitting device provided on the aircraft or present in a flight space of the aircraft.

[0129] REFERENCE SIGNS LIST 1 drone, 11 RGB camera, 12 stereo camera, 13 direction identification light, 14 projector, 100 information processing unit, 110 imaging device, 120 light emitting device, 131 imaging control unit, 132 light emitting control unit, 133 calculation processing unit, 134 flight control unit, 135 interference determination unit, 211 application processor, 212 vision sensing system, 213 flight controller, 214 communication module, 215 integrated controller, 221 infrared ranging sensor, 250 external light emitting device

Claims

1. An information processing device having a light-emitting control unit that switches control of a light-emitting device based on the exposure timing of a photographing device mounted on an aircraft and the setting information of a light-emitting device installed on the aircraft or present in the flight space of the aircraft.

2. The information processing device according to claim 1, wherein the setting information includes at least one of the light emission on / off cycle of the light emitting device, the light emission wavelength band, the positional relationship with the image capturing device, and the light emission range.

3. The information processing device according to claim 2, wherein the light emission control unit switches on / off the light emission of the light emitting device in the light emission wavelength band that can be photographed by the photographing device, according to the exposure timing.

4. The information processing device according to claim 3, wherein the light emission control unit does not control the light emission of the light emitting device in the emission wavelength band in which the imaging device cannot capture images.

5. The information processing device according to claim 2, wherein the light emission control unit does not control the positional relationship or light emission of the light emitting device in the light emission range where the light emission is not included in the shooting range of the shooting device.

6. The information processing device according to claim 1, further comprising an interference determination unit that determines whether the light emitted by the light emitting device is included in the imaging range of the imaging device, and generates an interference signal indicating that the light emitted by the light emitting device is interfering with imaging by the imaging device.

7. The information processing device according to claim 6, wherein the interference determination unit determines whether the light emitted by the light-emitting device is included in the imaging range based on the viewing angle of the imaging device and the light-emitting range of the light-emitting device, or based on the sensitivity band of the imaging device and the light-emitting wavelength band of the light-emitting device.

8. The information processing device according to claim 6, wherein the interference determination unit assigns the interference signal to an image captured by the imaging device during a period when the light emitted by the light emitting device is included in the imaging range.

9. The information processing device according to claim 8, further comprising an arithmetic processing unit that performs predetermined arithmetic processing based on the image captured by the imaging device, wherein the arithmetic processing unit performs the arithmetic processing by excluding the image to which the interference signal has been added.

10. The information processing device according to claim 1, wherein the light emitting device has a projection unit capable of projecting a predetermined pattern.

11. The information processing device according to claim 10, wherein an image including the pattern captured by the imaging device is used for analyzing an object onto which the pattern is projected.

12. The information processing device according to claim 10, further comprising an arithmetic processing unit that executes predetermined arithmetic processing based on an image including the pattern photographed by the photographing device.

13. The information processing device according to claim 12, wherein the calculation processing includes a self-position estimation process for the flying object.

14. The information processing device according to claim 1, wherein the exposure timing is determined based on the shutter speed of the image capturing device.

15. The information processing device according to claim 1, wherein the exposure timing is determined based on a flight plan for the flying object.

16. The information processing device according to claim 1, wherein the exposure timing is determined based on a photographing trigger input to the photographing device.

17. The information processing device according to claim 1, wherein the light-emitting control unit switches control of the light-emitting devices based on the exposure timing of a designated priority photographing device among the plurality of photographing devices mounted on the aircraft and the setting information of the plurality of light-emitting devices.

18. The information processing device according to claim 17, wherein the light emission control unit switches on / off the light emission of the desired light emitting device among the plurality of light emitting devices so that only the light emitted by the desired light emitting device is captured by the priority capturing device.

19. The information processing device according to claim 1, wherein the image capturing device is configured as one of an RGB camera, a stereo camera, an infrared camera, and a depth camera.

20. An information processing method including switching the control of a light-emitting device based on the exposure timing of a photographing device mounted on an aircraft and the setting information of a light-emitting device installed on the aircraft or present in the flight space of the aircraft.

Images