Information processing device, information processing method, program, mobile body, and system

Abstract

The present disclosure relates to an information processing device, an information processing method, a program, a mobile body, and a system that enable a plurality of mobile bodies to operate in a highly coordinated manner. An observation information acquisition unit recognizes another mobile body in an image that is obtained by a sensor device, acquires observation information indicating said other mobile body in the field of view of the sensor device, and specifies the position of said other mobile body on the image. An optical wireless communication information acquisition unit detects, in the image that is obtained by the sensor device, blinking of light with which said other mobile body outputs optical wireless communication information, acquires optical wireless communication information that is transmitted from said other mobile body in the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image. An action determination unit associates said other mobile body with the optical wireless communication information in accordance with the position of said other mobile body on the image and the position of the optical wireless communication information on the image and determines an action on the basis of at least the observation information and the optical wireless communication information. The present technology can be applied to, for example, a multi-agent system.

Description

Information processing device, information processing method, program, mobile object, and system

[0001] The present disclosure relates to an information processing device, an information processing method, a program, a mobile body, and a system, and in particular to an information processing device, an information processing method, a program, a mobile body, and a system that enable multiple mobile bodies to operate in a highly coordinated manner.

[0002] In a multi-agent system consisting of multiple agents (mobile entities) that act based on autonomous decisions, it has become necessary to identify each individual agent when these agents cooperate with each other.

[0003] For example, Patent Literature 1 discloses a measurement system that performs movement control for moving a work mobile body along a travel path based on the work mobile body's own position calculated by communication between the work mobile body and a support mobile body while the work mobile body and a support mobile body are alternately stationary. Patent Literature 2 discloses an optical communication system that acquires the position of a visible light source in an image as the position of the mobile body, and acquires information based on invisible light from an invisible light source provided on the mobile body.

[0004] JP 2022-159693 A JP 2023-92335 A

[0005] In conventional multi-agent systems, if the agents cannot be distinguished by appearance, it is not easy to identify the individual agents, making it difficult to have them work together in a more sophisticated manner.

[0006] The present disclosure has been made in light of such circumstances, and enables a plurality of moving bodies to operate in a highly coordinated manner.

[0007] An information processing device of one aspect of the present disclosure includes an observation information acquisition unit that recognizes other moving objects in an image obtained by a sensor device, acquires observation information indicating the other moving objects within the field of view of the sensor device, and identifies the position of the other moving objects on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device where the other moving objects output optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving objects within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an action determination unit that associates the other moving objects with the optical wireless communication information in accordance with the position of the other moving objects on the image and the position of the optical wireless communication information on the image, and determines an action based on at least the observation information and the optical wireless communication information.

[0008] An information processing method or program of one aspect of the present disclosure includes recognizing other moving objects in an image obtained by a sensor device, acquiring observation information indicating the other moving objects within the field of view of the sensor device, and identifying the position of the other moving objects on the image; detecting flickering of light in the image obtained by the sensor device by which the other moving objects output optical wireless communication information, acquiring the optical wireless communication information transmitted from the other moving objects within the field of view of the sensor device, and identifying the position of the optical wireless communication information on the image; and associating the other moving objects with the optical wireless communication information according to the position of the other moving objects on the image and the position of the optical wireless communication information on the image, and determining an action based on at least the observation information and the optical wireless communication information.

[0009] A moving body according to one aspect of the present disclosure comprises a sensor device; an observation information acquisition unit that recognizes other moving bodies in an image obtained by the sensor device, acquires observation information indicating the other moving bodies within the field of view of the sensor device, and identifies the position of the other moving bodies on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device, where the other moving bodies output optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving bodies within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an information processing unit that associates the other moving bodies with the optical wireless communication information according to the position of the other moving bodies on the image and the position of the optical wireless communication information on the image, and determines behavior based on at least the observation information and the optical wireless communication information.

[0010] A system of one aspect of the present disclosure is composed of a sensor device, a plurality of moving bodies including an observation information acquisition unit that recognizes other moving bodies in an image obtained by the sensor device, acquires observation information indicating the other moving bodies within the field of view of the sensor device, and identifies the position of the other moving bodies on the image, an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device where the other moving bodies output optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving bodies within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image, and an information processing unit that associates the other moving bodies with the optical wireless communication information according to the position of the other moving body on the image and the position of the optical wireless communication information on the image, and determines an action based on at least the observation information and the optical wireless communication information, and the plurality of moving bodies have a common configuration.

[0011] In one aspect of the present disclosure, other moving bodies are recognized in an image obtained by a sensor device, observation information indicating other moving bodies within the field of view of the sensor device is obtained, and the position of the other moving bodies on the image is identified, flashing of light outputting optical wireless communication information by the other moving bodies is detected in the image obtained by the sensor device, the optical wireless communication information transmitted from the other moving bodies within the field of view of the sensor device is obtained, and the position of the optical wireless communication information on the image is identified, and the other moving bodies are associated with the optical wireless communication information according to the position of the other moving bodies on the image and the position of the optical wireless communication information on the image, and behavior is determined based on at least the observation information and the optical wireless communication information.

[0012] FIG. 1 is a diagram illustrating a configuration example of an embodiment of a drone system to which the present technology is applied. FIG. 2 is a diagram illustrating a plan view of a drone system composed of three drones. FIG. 3 is a diagram illustrating an example of a captured image and an EVS image. FIG. 4 is a block diagram illustrating a configuration example of a drone. FIG. 5 is a flowchart illustrating information processing. FIG. 6 is a diagram illustrating another configuration example of a sensor device. FIG. 7 is a diagram illustrating optical wireless communication information including speed information. FIG. 8 is a diagram illustrating an optical wireless communication output unit having a plurality of light-emitting units. FIG. 9 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied.

[0013] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0014] <Configuration Example of Drone System> FIG. 1 is a diagram illustrating a configuration example of an embodiment of a drone system that is an example of a multi-agent system to which the present technology is applied.

[0015] The drone system 11 shown in Figure 1 is composed of multiple drones 12 (in the example shown, three drones 12A to 12C), and each drone 12 has a common configuration and is equipped with an optical wireless communication output unit 21 and a sensor device 22.

[0016] The optical wireless communication output unit 21 is configured with a light-emitting unit such as an LED (Light Emission Diode) and outputs optical wireless communication information by wireless communication using light that blinks using a predetermined modulation method. For example, the optical wireless communication information can include ID (Identification) information for individually identifying the drone 12, directional information indicating the direction in which the drone 12 is facing, and observation information indicating objects observed from the drone 12, other drones 12, etc.

[0017] The sensor device 22 is configured with an image sensor and an EVS (Event-based Vision Sensor), and acquires an image obtained by capturing an image using the image sensor (hereinafter referred to as a captured image), and an image obtained by detecting a change in brightness as the occurrence of an event using the EVS (hereinafter referred to as an EVS image).

[0018] For example, the drone 12 recognizes objects and other drones 12 captured in the captured image by performing image recognition processing on the captured image acquired by the sensor device 22, and acquires observation information indicating objects and other drones 12 within the field of view of the sensor device 22. Furthermore, the drone 12 detects, from the EVS image acquired by the sensor device 22, blinking of light when the optical wireless communication output unit 21 of the other drones 12 outputs optical wireless communication information, and acquires optical wireless communication information transmitted from the other drones 12 within the field of view of the sensor device 22.

[0019] Here, the sensor device 22 of the drone 12 is configured so that the field of view of the image sensor and the field of view of the EVS coincide. Therefore, the imaging position, which is the position on the captured image of the other drone 12 recognized in the captured image acquired by the sensor device 22, substantially coincides with the detection position, which is the position on the EVS image of the light of the optical wireless communication information of the other drone 12 detected in the EVS image acquired by the sensor device 22. This allows the drone 12 to easily associate the other drone 12 recognized in the captured image acquired by the sensor device 22 with the optical wireless communication information of the other drone 12 detected in the EVS image.

[0020] Therefore, for example, when the drone 12 recognizes multiple other drones 12 in a captured image, it can identify the multiple other drones 12 based on the ID information included in the associated optical wireless communication information. Furthermore, the drone 12 can decide on actions in cooperation with the multiple other drones 12 based on the direction information and observation information included in the optical wireless communication information output from the multiple other drones 12.

[0021] Identification of other drones 12 and cooperation with other drones 12 will be described with reference to Figures 2 and 3 .

[0022] Fig. 2 is a plan view of the drone system 11 configured with three drones 12A to 12C as shown in Fig. 1. Fig. 3A shows an example of an image captured by the sensor device 22B of the drone 12B, and Fig. 3B shows an example of an EVS image captured by the sensor device 22B of the drone 12B.

[0023] 2, the direction of the field of view of each of the drones 12A to 12C is indicated by a fan shape with a dashed line, and the light of the optical wireless communication information from the optical wireless communication output unit 21A of the drone 12A and the optical wireless communication output unit 21C of the drone 12C toward the drone 12B is indicated by an arrow with a dashed line. Also, the ID information of the drone 12A is ID0, the ID information of the drone 12B is ID1, and the ID information of the drone 12C is ID2.

[0024] In the example shown in Fig. 2, drone 12A is placed on the left side of the field of view of drone 12B, drone 12C is placed on the right side of the field of view of drone 12B, and object Obj-1 is placed between drone 12A and drone 12C. Therefore, in this case, drone 12B acquires an image in which drone 12A is recognized on the left side, drone 12C is recognized on the right side, and object Obj-1 is recognized between drone 12A and drone 12C, as shown in A of Fig. 3.

[0025] Similarly, in drone 12B, as shown in B of Fig. 3, an EVS image is acquired in which the light output by optical wireless communication output unit 21A of drone 12A as optical wireless communication information is detected on the left side, and the light output by optical wireless communication output unit 21C of drone 12C as optical wireless communication information is detected on the right side. The optical wireless communication information output from optical wireless communication output unit 21A includes ID information: ID0, directional information: dir0, and observation information: Obj-1, ID2. The optical wireless communication information output from optical wireless communication output unit 21C includes ID information: ID2, directional information: dir2, and observation information: Obj-3, ID1.

[0026] Therefore, drone 12B can identify drone 12A recognized in the captured image according to ID information ID0 of the optical wireless communication information detected at a detection position on the EVS image that approximately corresponds to the imaging position of drone 12A, and associate the optical wireless communication information with drone 12A. Similarly, drone 12B can identify drone 12C recognized in the captured image according to ID information ID2 of the optical wireless communication information detected at a detection position on the EVS image that approximately corresponds to the imaging position of drone 12C, and associate the optical wireless communication information with drone 12C. Furthermore, drone 12B can estimate, for example, the relative positions of drone 12A and drone 12C based on drone 12A and drone 12C captured in the captured image.

[0027] Drone 12B can then determine that object Obj-1 and drone 12C (ID information: ID2) are observed from the relative position of drone 12A estimated from the captured image and from the direction of drone 12A based on the directional information included in the optical wireless communication information of drone 12A. Similarly, drone 12B can determine that object Obj-3 and drone 12B (ID information: ID1) are observed from the relative position of drone 12C estimated from the captured image and from the direction of drone 12C based on the directional information included in the optical wireless communication information of drone 12C.

[0028] 2, the drone 12B can determine that the object Obj-3, which is not located within the field of view of the sensor device 22B, is located within the field of view of the sensor device 22C from the position and direction of the drone 12C. Therefore, the drone 12B can determine the approximate location where the object Obj-3 is estimated to be located based on the optical wireless communication information of the drone 12C and decide on an action, i.e., cooperate with the drone 12C.

[0029] As described above, even when the drones 12 cannot be distinguished from one another by appearance, the drone system 11 can easily identify each drone 12 based on the captured images and EVS images acquired by the sensor device 22. Furthermore, in the drone system 11, each drone 12 can appropriately determine its behavior based on the optical wireless communication information of the other drones 12. Therefore, in the drone system 11, the multiple drones 12 can operate in a highly coordinated manner.

[0030] For example, the drone system 11 can be used for multi-agent tasks using multiple general-purpose robots for which accurate position information is difficult to obtain. The drone system 11 can identify the multiple general-purpose robots based on ID information included in the optical wireless communication information, allowing the multiple general-purpose robots to cooperate and share tasks, enabling efficient operation. The drone system 11 is also suitable for use in underwater search robots that perform searches underwater, where wireless communication using electromagnetic waves is difficult. Furthermore, if this multi-agent system includes multiple cameras (image sensors and EVSs) with limited fields of view and fixed positions, the cameras can be tilted or panned to track target objects such as agents and bring them into the camera's field of view.

[0031] The drone system 11 can also be used for 3D scanning using multiple drones 12. For example, even if a marker or the like for recognizing the ID of the drone 12 is displayed on the housing of the drone 12, it is expected that it will be difficult to recognize the ID using the marker or the like in a three-dimensional scanning operation. In response to this, the drone system 11 can identify the drones 12 according to the ID information contained in the optical wireless communication information, and can operate efficiently by coordinating and sharing the 3D scanning operation.

[0032] The drone system 11 can also be used in a collision avoidance system (DAA: Detect and Avoid) for multiple drones 12. For example, by using VLC (Visible Light Communication) by the drone system 11 in addition to radar and lidar (Light Detection and Ranging), which are implemented in current collision avoidance systems, collision avoidance can be diversified and safety can be improved.

[0033] FIG. 4 is a block diagram showing an example of the configuration of the drone 12.

[0034] As shown in FIG. 4 , the drone 12 is configured to include an optical wireless communication output unit 21 , a sensor device 22 , a direction information acquisition unit 23 , and an information processing unit 24 .

[0035] The optical wireless communication output unit 21 includes an LED 31 that serves as a light emitting unit, and outputs optical wireless communication information by controlling the lighting of the LED 31 .

[0036] The sensor device 22 is configured to have an image sensor 32 and an EVS 33 , and supplies the captured image acquired by the image sensor 32 and the EVS image acquired by the EVS 33 to the information processing unit 24 .

[0037] The direction information acquisition unit 23 acquires direction information indicating the direction in which the drone 12 is facing, for example, using a gyro sensor or an acceleration sensor, and supplies the information to the information processing unit 24.

[0038] The information processing unit 24 executes information processing for cooperative operation of the multiple drones 12. As shown in the figure, the information processing unit 24 is configured to include an observation information acquisition unit 41, an optical wireless communication information acquisition unit 42, an optical wireless communication information generation unit 43, and an action determination unit 44.

[0039] The observation information acquisition unit 41 recognizes objects and other drones 12 appearing in the captured image by performing image recognition processing on the captured image acquired by the image sensor 32. The observation information acquisition unit 41 then acquires observation information indicating objects and other drones 12 within the field of view of the image sensor 32, and supplies this information to the optical wireless communication information generation unit 43 and the action decision unit 44. Furthermore, the observation information acquisition unit 41 identifies the image capture positions of the other drones 12 on the captured image, and supplies this image capture position to the action decision unit 44.

[0040] The optical wireless communication information acquisition unit 42 detects blinking of light output by the optical wireless communication output unit 21 of the other drone 12 from the EVS image acquired by the EVS 33. Then, the optical wireless communication information acquisition unit 42 acquires the optical wireless communication information transmitted from the other drone 12 within the field of view of the EVS 33 and supplies the information to the action decision unit 44. Furthermore, the optical wireless communication information acquisition unit 42 identifies the detection position of the optical wireless communication information output from the other drone 12 on the EVS image and supplies the detection position to the action decision unit 44.

[0041] The optical wireless communication information generation unit 43 generates optical wireless communication information including the drone 12's own ID information, directional information supplied from the directional information acquisition unit 23, and observation information supplied from the observation information acquisition unit 41, and supplies it to the optical wireless communication output unit 21.

[0042] The behavior determination unit 44 associates the other drones 12 with the optical wireless communication information based on the imaging positions of the other drones 12 provided by the observation information acquisition unit 41 and the detected positions of the optical wireless communication information output from the other drones 12 provided by the optical wireless communication information acquisition unit 42, and identifies the other drones 12 based on the ID information included in the optical wireless communication information. The behavior determination unit 44 then determines the behavior of the drone 12 itself in accordance with a pre-set policy based on the direction information provided by the direction information acquisition unit 23, the observation information provided by the observation information acquisition unit 41, and the optical wireless communication information provided by the optical wireless communication information acquisition unit 42. For example, the behavior determination unit 44 is configured with policies that instruct tasks such as work to be performed cooperatively by multiple drones 12, the relative distance to be maintained to avoid collisions with objects and other drones 12, and the behavior (e.g., surveillance) regarding target objects. Therefore, the behavior determination unit 44 can determine the behavior of the drone 12 itself so as to share work with the other drones 12 and avoid collisions, including with objects observed by the other drones 12.

[0043] The drone system 11 equipped with the drones 12 configured as described above can easily associate optical wireless communication information with multiple drones 12 that are indistinguishable by appearance, thereby enabling each drone 12 to be identified. For example, in a configuration in which IDs are transmitted and received via wireless communication using electromagnetic waves, it is difficult to associate IDs with other drones 12 recognized in a captured image. In contrast, the drone system 11 can easily associate optical wireless communication information with other drones 12 based on the captured position of the other drones 12 in the captured image and the detected position of the optical wireless communication information output from the other drones 12 in the EVS image. This enables the drone system 11 to cooperate using the relationships between the drones 12.

[0044] Furthermore, because the drone system 11 detects optical wireless communication information output from other drones 12 using two-dimensional EVS images, it can detect optical wireless communication information without, for example, performing optical axis alignment. For example, in a configuration using a light-receiving element such as a photodiode, it is necessary to align the optical axis of the photodiode with the other drone 12 that outputs the optical wireless communication information, and if the optical axis is not aligned, it may not be possible to detect the optical wireless communication information. In contrast, the drone system 11 does not require optical axis alignment and can more reliably detect optical wireless communication information. Furthermore, in a configuration using a light-receiving element such as a photodiode, it is not possible to distinguish the direction, etc., when multiple optical wireless communication information signals are detected, whereas the drone system 11 can distinguish the direction, etc., of multiple optical wireless communication information signals using two-dimensional EVS images.

[0045] Furthermore, the drone system 11 can simultaneously acquire multiple pieces of optical wireless communication information detected in the EVS image. As a result, for example, when acquiring multiple pieces of optical wireless communication information in a time-division manner, the optical wireless communication information needs to be received in a stationary state, whereas the drone system 11 does not require the drone 12 to be in a stationary state to receive the optical wireless communication information.

[0046] The information processing executed by the information processing unit 24 will be described with reference to the flowchart shown in FIG.

[0047] For example, when the drone system 11 is activated, processing begins, and in step S11, the observation information acquisition unit 41 acquires observation information by performing image recognition processing on the captured image acquired by the image sensor 32, and supplies the observation information to the optical wireless communication information generation unit 43 and the action decision unit 44. Furthermore, the observation information acquisition unit 41 identifies the image capture position of another drone 12 on the captured image, and supplies the image capture position to the action decision unit 44.

[0048] In step S12, the optical wireless communication information acquisition unit 42 acquires optical wireless communication information transmitted from the other drones 12 in accordance with the blinking of light detected from the EVS image acquired by the EVS 33, and supplies the information to the action decision unit 44. Furthermore, the optical wireless communication information acquisition unit 42 identifies the detected position of the other drone 12 on the EVS image, and supplies the detected position to the action decision unit 44.

[0049] In step S13, the optical wireless communication information generation unit 43 generates optical wireless communication information including the ID information of the drone 12 itself, the direction information supplied from the direction information acquisition unit 23, and the observation information supplied in step S12 from the observation information acquisition unit 41. Then, the optical wireless communication information generation unit 43 supplies the optical wireless communication information to the optical wireless communication output unit 21, which outputs the optical wireless communication information.

[0050] In step S14, the behavior decision unit 44 associates the other drones 12 with the optical wireless communication information in accordance with the imaging positions of the other drones 12 supplied from the observation information acquisition unit 41 in step S11 and the detected positions of the other drones 12 supplied from the optical wireless communication information acquisition unit 42 in step S12. Then, the behavior decision unit 44 determines the behavior of the drone 12 itself in accordance with a preset policy, based on the direction information supplied from the direction information acquisition unit 23, the observation information supplied from the observation information acquisition unit 41 in step S11, and the optical wireless communication information supplied from the optical wireless communication information acquisition unit 42 in step S12.

[0051] In step S15, the information processing unit 24 determines whether a task, such as work performed cooperatively by multiple drones 12, has been completed.

[0052] If the information processing unit 24 determines in step S15 that the task is not completed, the process returns to step S11, and the same process is repeated thereafter. On the other hand, if the information processing unit 24 determines in step S15 that the task is completed, the process ends.

[0053] By performing the above-described information processing in the information processing unit 24 of each drone 12, the drone system 11 allows multiple drones 12 to operate in a highly coordinated manner.

[0054] <Another Configuration Example of Sensor Device> Another configuration example of the sensor device 22 will be described with reference to FIG.

[0055] The sensor device 22 may have the above-described configuration including the image sensor 32 and the EVS 33, or may have only the image sensor 32 or only the EVS 33.

[0056] FIG. 6A shows an example of a captured image acquired by the sensor device 22B that is composed only of the image sensor 32B.

[0057] 6A, the drone 12B can visually recognize the drones 12A and 12C in the captured image acquired by the image sensor 32B, and can detect the light output by the optical wireless communication output unit 21A and the optical wireless communication output unit 21C as optical wireless communication information. Therefore, the drone 12B can associate the optical wireless communication information generated by the light detected at the positions where the drones 12A and 12C are visually recognized with the drones 12A and 12C, respectively.

[0058] FIG. 6B shows an example of an EVS image acquired by a sensor device 22 composed only of an EVS.

[0059] 6B, the drone 12B can detect changes accompanying the movements of the drones 12A and 12C in the EVS images acquired by the EVS 33, and can also detect light output as optical wireless communication information from the optical wireless communication output units 21A and 21C. Therefore, the drone 12B can associate the optical wireless communication information based on light detected at positions where changes accompanying the movements of the drones 12A and 12C are detected with the drones 12A and 12C, respectively.

[0060] In this way, even if the sensor device 22 is configured to include only an image sensor 32 or an EVS 33, the drone 12 can easily associate optical wireless communication information with other drones 12.

[0061] In addition, for a mobile object that moves at high speed, such as the drone 12, it is preferable to acquire optical wireless communication information using EVS images with a higher frame rate.

[0062] <Optical Wireless Communication Information Including Speed ​​Information> With reference to FIG. 7, optical wireless communication information including speed information will be described.

[0063] As shown in Figure 7, the optical wireless communication information can include speed information indicating the speed of the drone 12 in addition to ID information and direction information.

[0064] In the example shown in Figure 7, the optical wireless communication information output from the optical wireless communication output unit 21A includes ID information: ID0, direction information: dir0, and speed information: Vel0, and the optical wireless communication information output from the optical wireless communication output unit 21C includes ID information: ID2, direction information: dir2, and speed information: Vel2.

[0065] In this way, by adding speed information to the optical wireless communication information, the drone system 11 can cause each drone 12 to learn to understand the speed of other drones 12 and take action to avoid collisions, thereby further improving safety.

[0066] <Optical Wireless Communication Output Unit Having Multiple Light Emitting Units> With reference to FIG. 8, the optical wireless communication output unit 21 having multiple light emitting units will be described.

[0067] For example, the optical wireless communication output unit 21 is not limited to being configured to have one light emitting unit, but may be configured to have a plurality of light emitting units.

[0068] The drone system 11 may employ, for example, an optical wireless communication output unit 21' configured with four light-emitting units arranged in a 2x2 pattern, and each drone 12 is aware of the positional relationship, such as the spacing, between the four light-emitting units of the other drones 12. This allows the drone 12 to measure the distance to the other drones 12 based on the positional relationship, such as the spacing, on the EVS image where the four light-emitting units of the optical wireless communication output unit 21' provided on the other drones 12 are detected.

[0069] 8, the drone 12B can measure the distance to the drone 12A based on the positional relationship on the EVS image of the four light-emitting elements of the optical wireless communication output unit 21A'. Similarly, the drone 12B can measure the distance to the drone 12C based on the positional relationship on the EVS image of the four light-emitting elements of the optical wireless communication output unit 21C'.

[0070] Therefore, a drone system 11 that uses an optical wireless communication output unit 21 having multiple light-emitting elements can operate multiple drones 12 in a more highly coordinated manner by using distance in addition to the various pieces of information obtained by optical wireless communication information.

[0071] In the present embodiment, the drone system 11 has been described as an example of a multi-agent system as described above, but the present technology can be applied to, in addition to the drone 12, for example, self-driving vehicles and various robots (such as rovers) that can serve as agents that act based on autonomous decisions. Furthermore, in an environment where the drone system 11 can acquire position information of the drone 12 using, for example, a global positioning system (GPS) or a global positioning satellite, the position information may be included in the optical wireless communication information, and multiple drones 12 can operate in coordination using their respective position information.

[0072] Furthermore, in a multi-agent system to which the present technology is applied, captured images or EVS images containing optical wireless communication information can be processed directly in an image processing network to determine cooperative behavior among multiple agents. This allows a multi-agent system to perform more advanced cooperative behavior than, for example, multi-agent systems that determine behavior by processing captured images without communication, or multi-agent systems that acquire wireless communication information via wireless communication using electromagnetic waves and then process captured images to determine behavior.

[0073] <Example of Computer Configuration> Next, the above-described series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0074] FIG. 9 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0075] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an EEPROM (Electronically Erasable and Programmable Read Only Memory) 104 are interconnected by a bus 105. An input / output interface 106 is further connected to the bus 105, and the input / output interface 106 is connected to the outside.

[0076] In a computer configured as described above, the CPU 101 performs the above-described series of processes by loading programs stored in, for example, the ROM 102 and EEPROM 104 into the RAM 103 via the bus 105 and executing the programs. In addition, the programs executed by the computer (CPU 101) can be written in advance in the ROM 102, or can be installed or updated in the EEPROM 104 from outside via the input / output interface 106.

[0077] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0078] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0079] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0080] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0081] Furthermore, for example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0082] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0083] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0084] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0085] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0086] <Examples of Combinations of Configurations> The present technology may also be configured as follows: (1) An information processing device including: an observation information acquisition unit that recognizes another moving object in an image acquired by a sensor device, acquires observation information indicating the other moving object within a field of view of the sensor device, and identifies a position of the other moving object on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image acquired by the sensor device, where the other moving object outputs optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving object within a field of view of the sensor device, and identifies a position of the optical wireless communication information on the image; and an action determination unit that associates the other moving object with the optical wireless communication information in accordance with the position of the other moving object on the image and the position of the optical wireless communication information on the image, and determines an action based on at least the observation information and the optical wireless communication information. (2) The information processing device according to (1), wherein the sensor device is configured to have an image sensor that acquires captured images and an EVS (Event-based Vision Sensor) sensor that acquires EVS images in which a change in brightness is detected as the occurrence of an event, the observation information acquisition unit acquires the observation information by performing image recognition processing on the captured images and specifies a position of the other moving object on the captured images, and the optical wireless communication information acquisition unit acquires the optical wireless communication information from the EVS images and specifies a position of the optical wireless communication information on the EVS images. (3) The information processing device according to (2), wherein the optical wireless communication information includes at least ID (Identification) information for individually identifying the other moving object, and the behavior determination unit identifies the other moving object according to the ID information included in the optical wireless communication information specified at a position on the EVS image that approximately matches the position of the other moving object specified on the captured images. (4) The information processing device according to (3), wherein the optical wireless communication information includes at least the observation information, the ID information, and direction information indicating a direction in which the other moving body is facing, and the observation information further indicates an object within a field of view of the sensor device.(5) The information processing device according to (4), further comprising an optical wireless communication information generation unit that generates and outputs the optical wireless communication information including the observation information, the ID information for identifying its own moving body, and the direction information indicating a direction in which its own moving body is facing. (6) The information processing device according to any of (1) to (5), wherein the sensor device is configured to have only an image sensor that acquires a captured image, and the other moving body is recognized in the captured image and light of the optical wireless communication information is detected. (7) The information processing device according to any of (1) to (5), wherein the sensor device is configured to have only an EVS sensor that acquires an EVS image in which a change in luminance is detected as the occurrence of an event, and a change accompanying movement of the other moving body is detected in the EVS image and light of the optical wireless communication information is detected. (8) The information processing device according to (3), wherein the optical wireless communication information includes at least the ID information, direction information indicating a direction in which the other moving body is facing, and speed information indicating a speed of the other moving body. (9) The information processing device according to any one of (1) to (8) above, wherein the optical wireless communication information output unit that outputs the optical wireless communication information is provided with one or more light emitting units. (10) The information processing device according to any one of (1) to (9) above, wherein the information processing device is provided in a moving body, and the other moving body has a configuration common to the moving body.(11) An information processing method including an information processing device: recognizing another moving body in an image obtained by a sensor device, acquiring observation information indicating the other moving body within the field of view of the sensor device, and identifying the position of the other moving body on the image; detecting blinking of light in the image obtained by the sensor device, in which the other moving body outputs optical wireless communication information, acquiring the optical wireless communication information transmitted from the other moving body within the field of view of the sensor device, and identifying the position of the optical wireless communication information on the image; and associating the other moving body with the optical wireless communication information according to the position of the other moving body on the image and the position of the optical wireless communication information on the image, and determining an action based on at least the observation information and the optical wireless communication information. (12) A program for causing a computer of an information processing device to execute information processing including: recognizing other moving objects in an image obtained by a sensor device, acquiring observation information indicating the other moving objects within the field of view of the sensor device, and identifying the position of the other moving objects on the image; detecting blinking of light in the image obtained by the sensor device, where the other moving objects output optical wireless communication information, acquiring the optical wireless communication information transmitted from the other moving objects within the field of view of the sensor device, and identifying the position of the optical wireless communication information on the image; and associating the other moving objects with the optical wireless communication information according to the position of the other moving objects on the image and the position of the optical wireless communication information on the image, and determining an action based on at least the observation information and the optical wireless communication information.(13) A mobile object comprising: a sensor device; an observation information acquisition unit that recognizes other mobile objects in an image obtained by the sensor device, acquires observation information indicating the other mobile objects within the field of view of the sensor device, and identifies the position of the other mobile object on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device, in which the other mobile object outputs optical wireless communication information, acquires the optical wireless communication information transmitted from the other mobile object within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an action determination unit that associates the other mobile object with the optical wireless communication information according to the position of the other mobile object on the image and the position of the optical wireless communication information on the image, and determines an action based on at least the observation information and the optical wireless communication information. (14) A system comprising a sensor device; an observation information acquisition unit that recognizes other moving objects in an image obtained by the sensor device, acquires observation information indicating the other moving objects within the field of view of the sensor device, and identifies the position of the other moving object on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device, in which the other moving objects output optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving objects within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an action determination unit that associates the other moving objects with the optical wireless communication information according to the position of the other moving object on the image and the position of the optical wireless communication information on the image, and determines an action based on at least the observation information and the optical wireless communication information, wherein the multiple moving objects have a common configuration.

[0087] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0088] REFERENCE SIGNS LIST 11 Drone system, 12 Drone, 21 Optical wireless communication output unit, 22 Sensor device, 23 Direction information acquisition unit, 24 Information processing unit, 31 LED, 32 Image sensor, 33 EVS, 41 Observation information acquisition unit, 42 Optical wireless communication information acquisition unit, 43 Optical wireless communication information generation unit, 44 Action decision unit

Claims

1. An information processing device comprising: an observation information acquisition unit that recognizes other moving objects in an image obtained by a sensor device, acquires observation information indicating the other moving objects within the field of view of the sensor device, and identifies the position of the other moving object on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device where the other moving objects output optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving objects within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an action decision unit that associates the other moving objects with the optical wireless communication information in accordance with the position of the other moving object on the image and the position of the optical wireless communication information on the image, and decides an action based on at least the observation information and the optical wireless communication information.

2. The information processing device of claim 1, wherein the sensor device is configured to have an image sensor that acquires an imaged image, and an EVS sensor that acquires an EVS (Event-based Vision Sensor) image in which a change in brightness is detected as the occurrence of an event, the observation information acquisition unit acquires the observation information by performing image recognition processing on the captured image, and identifies the position of the other moving object on the captured image, and the optical wireless communication information acquisition unit acquires the optical wireless communication information from the EVS image, and identifies the position of the optical wireless communication information on the EVS image.

3. The information processing device described in claim 2, wherein the optical wireless communication information includes at least ID (Identification) information for individually identifying the other moving body, and the behavior decision unit identifies the other moving body according to the ID information included in the optical wireless communication information identified at a position on the EVS image that approximately coincides with the position of the other moving body identified on the captured image.

4. The information processing device of claim 3, wherein the optical wireless communication information includes at least the observation information, the ID information, and directional information indicating the direction in which the other moving body is facing, and the observation information further indicates an object within the field of view of the sensor device.

5. The information processing device according to claim 4, further comprising an optical wireless communication information generation unit that generates and outputs the optical wireless communication information including the observation information, the ID information for identifying the own moving body, and the directional information indicating the direction in which the own moving body is facing.

6. The information processing device according to claim 1, wherein the sensor device is configured to have only an image sensor for acquiring an image, and in the image, the other moving object is recognized and the light of the optical wireless communication information is detected.

7. The information processing device of claim 1, wherein the sensor device is configured to have only an EVS sensor that acquires an EVS image in which a change in brightness is detected as the occurrence of an event, and in the EVS image, changes associated with the movement of the other moving body are detected and light of the optical wireless communication information is detected.

8. The information processing device according to claim 3, wherein the optical wireless communication information includes at least the ID information, direction information indicating a direction in which the other moving body is facing, and speed information indicating a speed of the other moving body.

9. The information processing device according to claim 1, wherein the optical wireless communication information output unit that outputs the optical wireless communication information is provided with one or a plurality of light emitting units.

10. The information processing device according to claim 1, wherein the information processing device is provided in a moving body, and the other moving body has a common configuration with the moving body.

11. An information processing method comprising: an information processing device recognizing another moving object in an image obtained by a sensor device, acquiring observation information indicating the other moving object within the field of view of the sensor device, and identifying the position of the other moving object on the image; detecting blinking of light in the image obtained by the sensor device where the other moving object outputs optical wireless communication information, acquiring the optical wireless communication information transmitted from the other moving object within the field of view of the sensor device, and identifying the position of the optical wireless communication information on the image; and associating the other moving object with the optical wireless communication information according to the position of the other moving object on the image and the position of the optical wireless communication information on the image, and determining an action based on at least the observation information and the optical wireless communication information.

12. A program for causing a computer of an information processing device to execute information processing including: recognizing other moving objects in an image obtained by a sensor device, acquiring observation information indicating the other moving objects within the field of view of the sensor device, and identifying the position of the other moving object on the image; detecting blinking of light in the image obtained by the sensor device where the other moving object outputs optical wireless communication information, acquiring the optical wireless communication information transmitted from the other moving object within the field of view of the sensor device, and identifying the position of the optical wireless communication information on the image; and associating the other moving object with the optical wireless communication information according to the position of the other moving object on the image and the position of the optical wireless communication information on the image, and determining an action based on at least the observation information and the optical wireless communication information.

13. A mobile object comprising: a sensor device; an information processing unit having: an observation information acquisition unit that recognizes other moving objects in an image obtained by the sensor device, acquires observation information indicating the other moving objects within the field of view of the sensor device, and identifies the position of the other moving object on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device where the other moving object outputs optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving object within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an action decision unit that associates the other moving object with the optical wireless communication information in accordance with the position of the other moving object on the image and the position of the optical wireless communication information on the image, and decides an action based on at least the observation information and the optical wireless communication information.

14. A system comprising a sensor device; an observation information acquisition unit that recognizes other moving objects in an image obtained by the sensor device, acquires observation information indicating the other moving objects within the field of view of the sensor device, and identifies the position of the other moving object on the image; an optical wireless communication information acquisition unit that detects blinking of light in the image obtained by the sensor device where the other moving objects output optical wireless communication information, acquires the optical wireless communication information transmitted from the other moving objects within the field of view of the sensor device, and identifies the position of the optical wireless communication information on the image; and an action decision unit that matches the other moving objects with the optical wireless communication information in accordance with the position of the other moving object on the image and the position of the optical wireless communication information on the image, and decides an action based on at least the observation information and the optical wireless communication information, wherein the multiple moving objects have a common configuration.

Images