Sensing System and Sensing Method

Abstract

This sensing system comprises a mobile body on which a camera is mounted, and a plurality of sensor nodes installed at different installation sites within the sensing range, each of the sensor nodes comprising a sensor unit that detects sensor information at the installation site of the sensor node, an emitting unit that emits visible light, and a control unit that controls the emitting unit to emit the visible light in accordance with the sensor information. The mobile body moves to each of the sensor nodes successively, captures an image of the visible light emitted by the emitting unit at each of the sensor nodes with the camera, and performs visible light communication to acquire the sensor information.

Description

Technical Field

[0001] The disclosed technology relates to a sensing system and a sensing method.

Background Art

[0002] Among communications using optical signals, optical space communication (or optical wireless communication) is excellent in flexibility because it uses space as a transmission medium for optical signals instead of an optical fiber. This is also an alternative technology to wireless communication using radio waves, and in recent years when the shortage of frequency resources has become an issue, its application in IoT and the like is expected. In addition to conventional optical space communication using a laser or the like, in recent years, visible light communication using an LED has been actively studied. By using visible light such as an LED, there are merits such as cost reduction and flexible operation such as combined use with lighting applications such as ceiling lighting.

[0003] As a method of optical space communication using visible light, there is optical camera communication (OCC). OCC is visible light communication using a light source such as an LED or a display for a transmitter and a camera for a receiver. Using a three-color LED as a transmitter and modulating data into an optical signal for transmission is the most common application form. As for the operation on the receiving side, pixels occupied by the light source are extracted from a moving image captured by a camera, and a signal is demodulated from RGB values and the like in the area.

[0004] By applying OCC, in addition to smart devices equipped with an LED light and a camera such as a smartphone, communication can be performed using existing devices such as lighting and a web camera. That is, in addition to simple communication in daily life using a smartphone or the like, it is possible to apply it in various scenarios such as the coexistence of a lighting function and a communication function using an indoor light and V2X communication supporting autonomous driving.

[0005] As a related technology of OCC, there is a method (Non-Patent Document 1) for avoiding interference due to light diffusion called the Blooming effect. In OCC, an optical signal is received using a CMOS image sensor, and signal demodulation is performed using the RGB values of each pixel in the region corresponding to the light source in the image. At this time, if multiple light sources overlap on the image, interference occurs due to the overlap of light or the shielding of the light source itself. Depending on the model and environment, interference may further occur due to the light diffusion phenomenon called the Blooming effect. Regarding this problem, the technology of Non-Patent Document 1 formulates the interference conditions on the image theoretically by mathematical methods such as perspective transformation, taking into account the parameters of the camera such as the number of pixels, the size of the CMOS image sensor, and the angle of view, in addition to the coordinates and approximate size of the camera and the object.

[0006] Furthermore, it is possible to avoid interference between light sources by using an approximate formula representing the light attenuation on the image considering the Blooming effect.

[0007] Also, as a technique for improving the communication rate of OCC, there is a method called adaptive CSK (Color Shift Keying) (Non-Patent Document 2). Among the modulation methods of OCC, CSK that makes use of the characteristics of three-color LEDs has attracted attention from the perspective of throughput improvement. CSK is a modulation method in which the emission intensities of RGB three-color LEDs are changed to represent a plurality of colors such as 8 colors or 16 colors, and each color is associated with a bit string. For the color representation used in CSK, the color space defined by CIE1931 is used. CIE1931 is a quantitative color space that was first defined by the International Commission on Illumination between the wavelength distribution in the electromagnetic visible spectrum and the perceived color in human color vision. The constellation triangle is different depending on the imaging camera, and colors outside the triangle are corrected to the inside of the triangle. In CSK, communication is performed while preventing interference between symbols by providing symbols such that the distance is maximized within this triangle. However, in OCC, since the area occupied by the light source in the entire image is specified and the signal is demodulated from the RGB values within the area, depending on the color of the LED panel itself used as the light source, characteristics depending on the camera model, and external environments such as indoors or outdoors, it may be difficult to identify a specific color. To address this issue, Non-Patent Document 2 proposes adaptive CSK in which extra color symbols are prepared and the colors used for communication are changed according to the environment.

[0008] By preparing one more color symbol than in conventional CSK and performing data transmission in N colors excluding the color symbol with the lowest reading accuracy in the actual environment, it has been successful in improving the BER (Bit Error Rate) by up to 3 to 4 digits compared to normal CSK, and high-precision communication can be realized in various environments.

Prior Art Documents

Non-Patent Documents

[0009]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0010] In OCC, the transmission distance and transmission rate are restricted by factors such as the light source size, modulation method, and camera resolution. There has been no technology for constructing an efficient one-to-many communication system for receiving data from multiple transmission light sources using a single camera while considering such restrictions. That is, a method of controlling a camera-mounted device according to the arrangement of light sources has not been conventionally studied.

[0011] The disclosed technology aims to provide a sensing system and a sensing method capable of collecting sensor information from a plurality of sensor nodes using visible light communication.

Means for Solving the Problems

[0012] A first aspect of the present disclosure is a sensing system, which includes a mobile body equipped with a camera, a plurality of sensor nodes installed at different installation locations within a sensing range, and an information processing device. Each of the sensor nodes includes a sensor unit that detects sensor information at the installation location of the sensor node, an irradiation unit that irradiates visible light, and a control unit that controls the irradiation unit to irradiate visible light according to the sensor information. The mobile body sequentially moves to each of the sensor nodes, performs visible light communication to acquire the sensor information by photographing the visible light irradiated by the irradiation unit of each of the sensor nodes with the camera. The information processing device determines, for each of the plurality of sensor nodes, a maximum transmission time required for visible light communication of the sensor information of the sensor node based on the transmission rate of the visible light communication and the maximum data size of the sensor information, and outputs an operation command so that the mobile body photographs the visible light with the camera for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes.

[0013] A second aspect of the present disclosure is a sensing method, which is a sensing method in a sensing system including a mobile body equipped with a camera, a plurality of sensor nodes provided at different installation locations within a sensing range, and an information processing device. The sensor unit of each of the sensor nodes detects sensor information at the installation location of the sensor node, and the control unit controls an irradiation unit that irradiates visible light so as to irradiate visible light according to the sensor information. The mobile body sequentially moves to each of the sensor nodes, performs visible light communication to acquire the sensor information by photographing the visible light irradiated by the irradiation unit of each of the sensor nodes with the camera. The information processing device determines, for each of the plurality of sensor nodes, a maximum transmission time required for visible light communication of the sensor information of the sensor node based on the transmission rate of the visible light communication and the maximum data size of the sensor information, and outputs an operation command so that the mobile body photographs the visible light with the camera for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes.

Advantages of the Invention

[0014] According to the disclosed technology, sensor information can be collected from a plurality of sensor nodes using visible light communication.

Brief Description of the Drawings

[0015]

Figure 1A

Figure 1B

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the disclosed technology will be described in detail with reference to the drawings. An example will be described in which the disclosed technology is applied to a sensing system that controls a flying object to fly so as to collect sensor information obtained by a plurality of sensor nodes installed at different locations in a farm.

[0017] <Configuration of Sensing System 10> As shown in FIG. 1A, a sensing system 10 according to an embodiment of the disclosed technology includes an information processing device 18, a flying object 20 that flies within a sensing range, and a plurality of sensor nodes 40 provided at different locations within the sensing range. In an embodiment of the present invention, the sensing range is, for example, a farm. The flying object 20 and the information processing device 18 are connected by wireless communication.

[0018] The sensor node 40 includes an irradiation unit 42, a control unit 44, and a sensor unit 46. The sensor unit 46 detects sensor information about a desired target at the installation location of the sensor node using some sensor. For example, temperature, humidity, illuminance, or soil moisture is detected as sensor information. Note that the type of sensor information is not limited to this, and any type of sensor information may be used, such as the atmospheric pressure measured by a barometer or the temperature distribution measured by an infrared sensor. Also, a plurality of sensor units 46 may be provided. FIG. 1A shows an example in which each sensor node 40 includes two sensor units 46A and 46B, two control units 44A and 44B, and two irradiation units 42A and 42B.

[0019] In addition to a general three-color LED, the irradiation unit 42 irradiates visible light using a panel in which a plurality of LEDs are arranged, a display, or the like. The control unit 44 controls so as to transmit an optical signal by the light emission of the irradiation unit 42. As a modulation method, in addition to a method called On Off Keying that represents 1 and 0 by turning the light emission on and off, a method such as Color Shift Keying that represents a bit string by the color of the emitted light can be used. Also, a method called spatial modulation using a plurality of light sources may be used. In this way, the control unit 44 plays a role of converting the sensor information acquired by the sensor unit 46 into an optical signal using a specified modulation method.

[0020] Light sources used in the irradiation unit 42 include an LED light, an LED panel, a display, and the like. The sensor information detected by the sensor unit 46 is encoded and modulated as an optical signal and transmitted from the light source.

[0021] A plurality of sensor nodes 40 are installed within the sensing range. The position of the sensor node 40 may be arbitrarily determined according to the object to be sensed. However, the position of the sensor node 40 is assumed to be known. The position of the sensor node 40 may be measured in advance using GPS or the like. Each sensor node 40 transmits the sensor information acquired using the sensor unit 46 as an optical signal using the irradiation unit 42. Regarding the transmission time of the sensor information, it may emit light constantly, or may emit light at a predetermined time.

[0022] As shown in FIG. 1B, the control unit 44 includes a sensor information acquisition unit 70, a proximity detection unit 72, a sensor node number transmission control unit 74, and a sensor information transmission control unit 76.

[0023] The sensor information acquisition unit 70 acquires the sensor information detected by the sensor unit 46. The approach detection unit 72 detects the approach of the flying object 20. Specifically, the sensor unit 46 includes an infrared sensor and detects the infrared rays from the flying object 20 to detect the approach of the flying object 20. Note that the flying object 20 may further include a light irradiation unit, and the sensor unit 46 may receive the light from the light irradiation unit of the flying object 20 to detect the approach of the flying object 20.

[0024] When the approach detection unit 74 detects the approach of the flying object 20, it starts the light emission by the irradiation unit 42, and at this time, controls the light emission by the irradiation unit 42 to transmit the identification number for identifying the sensor node 40.

[0025] After the sensor information transmission control unit 76 transmits the identification number of the sensor node 40, it controls to transmit the sensor information acquired using the sensor unit 46 as an optical signal using the irradiation unit 42.

[0026] As shown in FIG. 2, the flying object 20 includes a flying body unit 21, arms 22, and four rotary wings 23. The flying body unit 21 is provided at the central part of the flying object 20 in plan view, and a communication antenna, a battery, a photographing unit 26, etc. (not shown) are mounted on the flying body unit 21.

[0027] Four arms 22 are provided and radially protrude outward from the flying body unit 21 in plan view. A rotary wing 23 is attached to the tip of each of the arms 22. The arm 22 is hollow, and a motor for operating the rotary wing 23, control and power wiring, etc. are housed inside.

[0028] A photographing unit 26 is mounted on the lower surface side of the flying body unit 21. The photographing unit 26 is installed so as to be able to capture an image of the sensor node 40. The photographing unit 26 has parameters such as the number of pixels, the viewing angle, the frame rate, and the exposure amount.

[0029] The flight body 21 is equipped with a flight control device 30 shown in Fig. 3A. The flight control device 30 is composed of a CPU, a ROM, a RAM, ports for input and output of signals transmitted and received by a communication antenna, etc., and stores various processing programs for controlling the flying object 20, data, etc.

[0030] The flight control device 30 controls the rotary wings 23 to recognize the position of its own aircraft and fly automatically along the designated movement route. Functionally, the flight control device 30 is configured as follows. As shown in Fig. 3A, the flight control device 30 includes a communication unit 32, an autonomous attitude control unit 34, a position measurement unit 36, a control unit 38, and a sensor information storage unit 39.

[0031] The communication unit 32 transmits and receives signals to and from the information processing device 18 by wireless communication.

[0032] The autonomous attitude control unit 34 controls the four rotary wings 23 to perform hovering.

[0033] The position measurement unit 36 measures the current position of the flying object 20 using a GPS sensor (not shown) provided on the flying object 20.

[0034] As shown in Fig. 3B, the control unit 38 includes a movement control unit 80, a shooting control unit 82, a sensor node detection unit 84, and a sensor information acquisition unit 86. Based on the current position measured by the position measurement unit 36, the movement control unit 80 sequentially moves to each of the sensor nodes 40 along the movement route and controls the autonomous attitude control unit 34 to fly automatically.

[0035] In addition, the imaging control unit 82 controls the imaging unit 26 to image each of the sensor nodes 40 that are traveling. Specifically, when the flying object 20 moves to a position corresponding to the installation location of the sensor node 40, the control unit 38 controls as follows. Based on the position of the sensor node 40 and the current position of the flying object 20, the control unit 38 turns the imaging direction of the imaging unit 26 toward the sensor node 40, and the imaging unit 26 images the visible light irradiated by each irradiation unit 42 of the sensor node 40 for a predetermined time. Here, the control may be such that the orientation of the imaging unit 26 is changed so as to direct the imaging direction of the imaging unit 26 toward the sensor node 40, or the control may be such that the orientation of the flying object 20 is changed.

[0036] The sensor node detection unit 84 detects, from the video imaged by the imaging unit 26, a region representing the visible light irradiated by each irradiation unit 42 of the sensor node 40. The sensor information acquisition unit 86 demodulates the sensor information from the region representing the visible light in the video imaged by the imaging unit 26 and stores it in the sensor information storage unit 39.

[0037] <Configuration of information processing device> The information processing device 18 is configured by a computer including a CPU, a ROM, a RAM, and an HDD.

[0038] Functionally, the information processing device 18 is configured as follows. As shown in FIG. 4, the information processing device 18 includes a communication unit 50, an acquisition unit 52, a route determination unit 54, a flight control unit 56, a collection unit 58, and a sensor information storage unit 60.

[0039] The communication unit 50 receives sensor information from the flying object 20 by wireless communication.

[0040] The acquisition unit 52 acquires the transmission rate of visible light communication and the maximum data size of the sensor information input by the user.

[0041] The route determination unit 54 determines a movement route and a movement speed for sequentially moving to each of the sensor nodes 40. At this time, the route determination unit 54 obtains the maximum transmission time required for visible light communication of the sensor information based on the transmission rate of visible light communication and the maximum data size of the sensor information. The route determination unit 54 determines the movement route and the movement speed so that the aircraft 20 moves to capture visible light by the imaging unit 26 for a time equal to or longer than the maximum transmission time at the sensor node 40.

[0042] Specifically, for each of the plurality of sensor nodes 40, the route determination unit 54 obtains the maximum transmission time required for visible light communication of the sensor information based on the transmission rate of visible light communication and the maximum data size of the sensor information of the sensor node 40. The route determination unit 54 determines the movement route and the movement speed so that the aircraft 20 moves to capture visible light by the imaging unit 26 for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes 40. More specifically, the route determination unit 54 determines the movement route and the movement speed so that the aircraft 20 stays at a position where visible light can be captured by the imaging unit 26 for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes 40. Thereby, the aircraft 20 hovers so as to stay at a position and posture where visible light can be captured by the imaging unit 26 for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes 40.

[0043] Also, even when the types of sensor information are different for each sensor node 40, the aircraft 20 can hover so as to stay at a position and posture where visible light can be captured by the imaging unit 26 for a time equal to or longer than the maximum transmission time corresponding to the type of sensor information at the sensor node 30.

[0044] Incidentally, as described above, when the flying object 20 is hovering and the imaging unit 26 captures the visible light irradiated by the irradiation unit 42 of the sensor node 40 to receive sensor information, if it is detected that due to a bird crossing or a gust of wind, etc., the sensor node 40 temporarily moves out of the imaging range of the imaging unit 26, resulting in some missing visible light communication within the transmission time, then the hovering can be continued so that the imaging unit 26 captures the visible light irradiated by the irradiation unit 42 of the sensor node 40 again to receive complete sensor information.

[0045] The collection unit 58 collects the sensor information of each sensor node 40 received from the flying object 20 by wireless communication and stores it in the sensor information storage unit 60.

[0046] The flight control unit 56 transmits an operation command instructing the flying object 20 to move at the determined movement route and movement speed to the flying object 20 via the communication unit 50.

[0047] Thereby, the flying object 20 moves while capturing a moving image using the imaging unit 26 according to the movement route and movement speed of the operation command (see FIG. 5). At this time, even when the external environment (for example, wind speed, wind direction) is different at each of the sensor nodes 40, the flying object 20 can perform hovering so as to stay at a position and posture where the imaging unit 26 can capture visible light for a time longer than the maximum transmission time for the relevant sensor node 30. The imaging unit 26 captures the optical signal emitted by the irradiation unit 42 of the sensor node 40 and receives sensor information by demodulating the optical signal from the captured moving image. Note that the demodulation of the captured moving image may be performed by the flying object 20 or by the information processing device 18.

[0048] Next, with reference to FIG. 6, the extraction of the irradiation unit 42 on the moving image captured by the imaging unit 26 and the reading of the optical signal will be described.

[0049] Define the x-axis and y-axis as the image coordinate system on the moving image captured by the imaging unit 26 (see Fig. 6). The ranges of x and y are determined by the resolution. The pixels occupied by the irradiation unit 42 on the image are represented as image coordinates, and these can be easily mutually converted with the real-world coordinates by using parameters such as the angle of view of the imaging unit 26. In a certain frame, identify and extract the pixel region occupied by the irradiation unit 42, and denote this as region k (k = 1, 2,...). The method for extracting the pixel region can be any method. For example, the image coordinates can be calculated and extracted from the real-world coordinates and orientation of the imaging unit 26 of the flying object 20 and the coordinates of the sensor node 40, or dynamic extraction can be performed by mechanically detecting a specific form or the like using techniques such as deep learning.

[0050] For each region k, totalize the RGB values within the region. Regarding the totalization method, in addition to taking the average value of the RGB values within the region, the distribution of the values for each pixel can also be used. Demodulate the signal based on the totalized values. At this time, the demodulation method depends on the method at the time of modulation.

[0051] Also, as shown in Fig. 6, the moving speed v x , v y of the irradiation unit 42 in the image coordinate system is defined. These values can be calculated from the coordinates, moving direction, and speed of the flying object 20 in the real world, the resolution and angle of the imaging unit 26, and the coordinates of the sensor node 40. And the existence time of the irradiation unit 42 in the image, that is, the time from when the irradiation unit 42 appears in the image until it disappears due to the movement of the flying object 20, can be calculated from the appearance position of the irradiation unit 42 and the moving speed v x , v y of the irradiation unit 42 in the image coordinate system. Then, from the transmission bit rate from the irradiation unit 42 determined by the modulation method and the existence time of the irradiation unit 42, the maximum transmission data volume is determined.

[0052] Since the actual transmission data volume is determined from the types of data to be sensed and the required conditions of the system, the sensing system is configured by determining the moving route and moving speed of the flying object 20 so that the actual transmission data volume does not exceed the maximum transmission data volume.

[0053] To efficiently collect sensor information, the route determination algorithm of the aircraft 20 is used. In the route determination algorithm, the movement route and movement speed are calculated based on the position and size of the irradiation unit 42 and the specifications of the imaging unit 26. The route determination algorithm will be described in detail below.

[0054] (Coordinate system) First, the coordinate system used in this embodiment is shown in FIG. 7. The global coordinate system is defined as (x, y, z). Also, the camera coordinate system is defined as (X, Y, Z). The Y axis corresponds to the center line of the image. Note that the origin is set at (x c (t), y c (t), z c (t)) of the imaging unit 26. The elevation angle of the imaging unit 26 is defined as θ.

[0055] (Variable definition) The variables used in this embodiment are summarized in Table 1.

[0056]

Table 1

[0057] Here, let I be the set of sensor nodes 40, and i and j be the identifiers of the sensor nodes 40. Also, let the position of the i-th sensor node 40 in the global coordinate system be (x i , y i , z i ). For simplicity, the sensor node 40 is approximated by a sphere with a radius of r i . Let (u i , v i ) indicate the coordinates of the center of the i-th sensor node 40 in the image plane. Let the horizontal resolution of the image be l h , and the vertical resolution be l v . Also, let the horizontal field angle and vertical field angle of the camera be denoted as φ h and φ v . Let the focal length of the imaging unit 26 be denoted as f. Also, define the image sensor size as image ρ.

[0058] (Coordinate Transformation) The position of the sensor node 40 on the image plane is calculated by coordinate transformation. Since a moving flying object 20 is assumed, the relative position between (x i , y i , z i ) and the origin (x c (t), y c (t), z c (t)) is calculated. The position of the i-th sensor node 40 in the camera coordinate system is obtained by using translation and rotation by the elevation angle as JPEG0007706200000002.jpg23105 (1) formulated as follows.

[0059] And the position of the i-th sensor node 40 in the image plane is calculated by perspective transformation as JPEG0007706200000003.jpg24140 (2) formulated as follows.

[0060] The perspective transformation is depicted as shown in Figure 7. Let p i represent the size of the sensor node 40 in the image plane. Let p i be calculated using the radius of the projection circle as JPEG0007706200000004.jpg1743 (3) formulated as follows.

[0061] The conditional expression that the i-th sensor node 40 is included in the photographable range by the photographing unit 26 of the flying object 20 is JPEG0007706200000005.jpg2077 (4) formulated as follows.

[0062] (Transmission Time) Here, the transmission time of sensor information from the sensor node 40 is formulated. The transmission rate of OCC is determined by the modulation number and the symbol rate. In order to increase the data transfer speed, optical spatial modulation and color shift keying (CSK) are adopted. In optical spatial modulation, a plurality of irradiation units 42 are used. CSK utilizes the design of a three-color lighting fixture of an LED. The optical signal is intensity-modulated to generate predefined constellation symbols (see Non-Patent Document 2). The range of the symbol rate is limited by the frame rate of the imaging unit 26 and the image processing speed.

[0063] R i Let R be the data rate (transmission rate), S i be the optical spatial multiplexing, D be the symbol rate, and N i be the number of constellation symbols. Then, JPEG0007706200000006.jpg1156 (5) is formulated as follows.

[0064] The maximum transmission time T i is defined with the maximum data size being M i as, JPEG0007706200000007.jpg2132 (6) is calculated as follows.

[0065] (Trajectory Requirement) The imaging unit 26 of the flying object 20 needs to image each sensor node 40 for a sufficient time to receive sensor information. Here, let τ i be the length of time that satisfies Equation (4). To surely receive the maximum data size from the i-th sensor node 40, τ i should JPEG0007706200000008.jpg1229 (7) satisfy the following.

[0066] (Route Determination Algorithm) In the route determination algorithm of this embodiment, an approximate shortest route that guarantees data transmission from all sensor nodes 40 is determined. Specifically, the route determination algorithm of this embodiment includes the following steps a to c.

[0067] (Step a) Clustering of sensor nodes The sensor nodes 40 are clustered based on the installation locations of the sensor nodes 40, the sensor nodes 40 included in each cluster obtained by clustering are grouped, and virtual sensor nodes 140 representing each cluster are generated (see FIG. 8). Adjacent sensor nodes 40 are grouped so as to satisfy the condition of being included in the photographable range of the above formula (4).

[0068] (Step b) Graph generation A graph G = (V, E) is generated, which consists of vertices representing the virtual sensor nodes 140 and edges connecting the virtual sensor nodes 140.

[0069] (Step c) Determination of movement route and movement speed The movement route is calculated by solving the traveling salesman problem (TSP) using the graph G. Also, the movement speed is determined so that the photographing time of each virtual sensor node 140 satisfies the above formula (6).

[0070] <Operation of the sensing system 10> Next, the operation of the sensing system 10 according to the embodiment of the present invention will be described.

[0071] First, the user inputs, to the information processing device 18, as information regarding each sensor node 40, the installation location of each sensor node 40, the transmission rate of visible light communication, and the maximum data size of sensor information. Then, the information processing device 18 executes the route determination processing routine shown in FIG. 9.

[0072] In step S100, the acquisition unit 52 acquires the transmission rate of visible light communication and the maximum data size of sensor information input by the user.

[0073] In step S102, the route determination unit 54 obtains the maximum transmission time required for visible light communication of the sensor information for each sensor node 40 based on the transmission rate of visible light communication and the maximum data size of the sensor information.

[0074] In step S104, the route determination unit 54 determines the movement route and movement speed based on the installation location of each sensor node 40 and the maximum transmission time at each sensor node 40. Specifically, the route determination unit 54 determines the movement route and movement speed for the flying object 20 to move sequentially to each of the sensor nodes 40 while capturing visible light by the imaging unit 26 for a time equal to or longer than the maximum transmission time at each sensor node 40.

[0075] In step S106, the flight control unit 56 transmits an operation command instructing the flying object 20 to move at the determined movement route and movement speed to the flying object 20 via the communication unit 50.

[0076] Then, when the communication unit 32 of the flying object 20 receives the operation command, the flying object 20 moves while capturing a moving image using the imaging unit 26 according to the movement route and movement speed of the operation command.

[0077] Then, in the sensor node 40, the sensing processing routine shown in FIG. 10 is executed by the control unit 44.

[0078] First, in step S110, the control unit 44 acquires the sensor information of the installation location of the sensor node 40 detected by the sensor unit 46.

[0079] In step S112, the control unit 44 converts the acquired sensor information into an optical signal using a specified modulation method.

[0080] In step S114, the control unit 44 controls the irradiation unit 42 to transmit the optical signal by emitting light.

[0081] The above step S114 is realized by the processing routine shown in FIG. 1.

[0082] First, in step S120, the approach detection unit 72 determines whether it has detected the approach of the flying object 20 using a sensor such as an infrared sensor. If the approach of the flying object 20 is detected, the process proceeds to step S122.

[0083] In step S122, the sensor node number transmission control unit 74 starts the light emission by the irradiation unit 42 and controls the light emission by the irradiation unit 42 to transmit an identification number for identifying the sensor node 40.

[0084] In step S124, the sensor information transmission control unit 76 controls to transmit the sensor information acquired using the sensor unit 46 as an optical signal using the irradiation unit 42.

[0085] In step S126, the sensor information transmission control unit 76 controls the light emission by the irradiation unit 42 to notify the end of the sensor information.

[0086] In step S128, the approach detection unit 72 determines whether the flying object 20 has left using a sensor such as an infrared sensor. If it is determined that the flying object 20 has not left, the process returns to the above step S122. On the other hand, if it is determined that the flying object 20 has left, the processing routine ends.

[0087] Also, the control unit 38 of the flying object 20 executes the movement control processing routine shown in FIG. 12.

[0088] First, in step S130, the movement control unit 80 acquires the movement route included in the operation command received from the information processing device 18.

[0089] In step S132, the movement control unit 80 moves along the movement route to a position where visible light communication can be performed with the next sensor node 40 based on the current position measured by the position measurement unit 36, and controls the autonomous attitude control unit 34 to fly automatically.

[0090] In step S134, the control unit 38 starts photographing, with the photographing direction of the photographing unit 26 facing the sensor node 40, the visible light irradiated by the irradiation unit 42 of the sensor node 40, based on the position of the sensor node 40 and the current position of the flying object 20.

[0091] In step S136, the sensor node detection unit 84 detects, from the video photographed by the photographing unit 26, a region representing the visible light irradiated by the irradiation unit 42 of the sensor node 40, and demodulates and acquires the sensor node number from the region representing the visible light in the video photographed by the photographing unit 26.

[0092] In step S138, the sensor information acquisition unit 86, the sensor node detection unit 84 detects, from the video photographed by the photographing unit 26, a region representing the visible light irradiated by the irradiation unit 42 of the sensor node 40. The sensor information acquisition unit 86 demodulates the sensor information from the region representing the visible light in the video photographed by the photographing unit 26 and stores it in the sensor information storage unit 39.

[0093] In step S140, the control unit 38 determines whether or not the processing in steps S132 to S138 has been completed for all the sensor nodes 40. If there is a sensor node 40 for which the processing in steps S132 to S138 has not been performed, it returns to step S132 and moves to a position where visible light communication can be performed with the sensor node 40. On the other hand, if the processing in steps S132 to S138 has been completed for all the sensor nodes 40, it proceeds to step S142.

[0094] In step S142, based on the current position measured by the position measurement unit 36, the movement control unit 80 controls the autonomous attitude control unit 34 to move along the movement route to the end point of the movement route and fly automatically.

[0095] Then, the communication unit 32 of the flying object 20 transmits the sensor information stored in the sensor information storage unit 39 to the information processing device 18 by wireless communication.

[0096] The collection unit 58 of the information processing device 18 collects the sensor information of each sensor node 40 received from the flying object 20 by wireless communication and stores it in the sensor information storage unit 60.

[0097] As described above, according to the sensing system according to the embodiment of the present invention, each of the sensor nodes detects the sensor information of the installation location of the sensor node and controls the irradiation unit to irradiate visible light according to the sensor information. The flying object sequentially moves to each of the sensor nodes, captures the visible light irradiated by the irradiation unit of each of the sensor nodes with a camera, and performs visible light communication to acquire sensor information. Thereby, sensor information can be collected from a plurality of sensor nodes.

[0098] In addition, a camera mounted on the flying object is used to efficiently receive the optical signals transmitted from a large number of scattered irradiation units. That is, based on the known arrangement of the irradiation units, the movement route and speed of the flying object are controlled according to parameters such as the irradiation unit size, modulation method, data size, and camera resolution. Thereby, it becomes possible to configure a sensing system that collects sensor information through visible light communication between the flying object and the optical camera. Since visible light communication does not use radio waves, it is possible to deploy a sensing system using a large number of sensors even in radio wave dead zones.

[0099] In addition, in this embodiment, a large number of sensor nodes 40 can be accommodated without being affected by interference in the time domain and frequency domain by OCC. Also, since no radio waves are used at all, it can be deployed even in places where the radio wave environment is quiet or the infrastructure is not well-developed.

[0100] In addition, a large number of sensor units are installed in the farm, and quasi-real-time environmental sensing is performed by collecting sensor information. By directly communicating between the sensor node and the flying object through visible light communication, a comprehensive sensor network can be configured even in an area where LTE / 5G or LPWA base stations are not well-equipped nearby.

[0101] In addition, in this embodiment, it is possible to configure a sensor network system that collects sensor information from the flying object through visible light communication. Since visible light communication does not use radio waves, it is possible to deploy a sensor network system using a large number of sensors even in radio wave dead zones.

[0102] Also, since imaging of crops by the imaging unit of the flying object can be performed simultaneously with sensing, it is possible to also perform sensing of pests, diseases, growth conditions, etc. from the state of leaves and the like.

[0103] In the above-described embodiment, the case where the aircraft measures its current position using a GPS sensor provided on the aircraft has been described as an example, but it is not limited thereto. For example, in order to identify the position of the sensor node in a short time, a beacon may be transmitted from the sensor node side and received on the aircraft side to confirm the position of the sensor node, and when the sensor node is identified by the imaging unit, imaging may be started. In that case, the sensor information communicated by visible light communication may be constantly lit or lit in a form that matches the time zone when the aircraft passes. In this case, the visible light communication may include a signal for identifying the start and end of communication of the sensor information, or the beacon may contain identification information of the sensor node. Further, the aircraft may estimate its position by a self-position estimation method from the position of the sensor node obtained in advance and the positional relationship of the sensor node on the image captured by the imaging unit. Specifically, the position coordinates of the aircraft are obtained using the relational expression of coordinate conversion between the real-world coordinates and the image coordinates. In particular, the more the number of irradiation units simultaneously captured on the image, the more accurate the positioning is possible. At this time, each irradiation unit may be made identifiable from the aircraft side by transmitting information such as a sensor node number given in advance together with the sensor information. Further, the current position of the aircraft measured using the GPS sensor may be corrected using the position of the aircraft estimated by the self-position estimation method. Further, each irradiation unit may further transmit the position information of the sensor node given in advance together with the sensor information. Further, the aircraft may estimate the relative position of the next sensor node from the current sensor node position and the next sensor node position and move to the next sensor node.

[0104] In addition, the case where the sensing range is a farm has been described as an example, but it is not limited thereto. For example, a radio wave dead zone other than a farm may be used as the sensing range. Representative examples of radio wave dead zones include underground spaces such as tunnels and caves, and underwater environments with high radio wave attenuation.

[0105] In the case of an underwater environment, as shown in FIG. 13, a submersible 220 such as an underwater drone may be used as the moving object. The submersible 220 is equipped with a photographing unit 26 in the same manner as the flying object 20. In this way, sensing in an environment where it was difficult to collect sensor information conventionally becomes possible. Also, in the submersible, GNSS does not function sufficiently, and it is difficult to use a GPS sensor. For this reason, the submersible estimates its position by a self-position estimation method based on the position of the sensor node required in advance and the positional relationship of the sensor node on the image captured by the photographing unit. Thereby, even in water where radio wave attenuation is large and wireless communication is difficult, wide-range sensing becomes possible, and it can be applied to environmental conservation, resource exploration, aquaculture, maintenance of port facilities, etc.

[0106] Also, in the case of underwater, if for some reason the visibility underwater is worse than expected, data transmission may not be completed within the time calculated based on the transmission rate and transmission distance input by the user in advance. In that case, continue photographing until it is detected that the data transmission has ended, or it may be possible to approach the light source more. Furthermore, the stored transmission rate may be corrected. As a means for detecting that the data transmission has ended, the light source can stop emitting light for a certain period of time, or emit a signal indicating the end of transmission.

[0107] Also, in the case of underwater, turbidity in the water may be detected from the image captured by the photographing unit 26, and the transmission rate and transmission distance may be corrected. At this time, the brightness of the light source is measured and compared with the assumed brightness of the light source based on the input transmission rate and transmission distance, thereby determining the turbidity in the water. If it is below a predetermined brightness, the transmission rate and transmission distance may be corrected.

[0108] Also, the sensing range may be indoors. In this case, as the moving body, a self-propelled moving body can be considered. Also, indoors, since the accuracy of the GPS sensor becomes low, the self-propelled moving body estimates the position of the self-propelled moving body by a self-position estimation method based on the position of the sensor node obtained in advance and the positional relationship of the sensor node on the image captured by the imaging unit. In addition, in order to specify the position of the sensor node in a short time, as described above, a beacon may be transmitted from the sensor node side.

[0109] Also, although the case where the flying body starts imaging by the imaging unit when it moves to a position corresponding to the installation location of the sensor node has been described as an example, it is not limited to this. While the flying body is moving, imaging may be continuously performed by the imaging unit.

[0110] Regarding the above embodiments, the following additional remarks are further disclosed.

[0111] [Appendix 1] A moving body equipped with a camera, A plurality of sensor nodes installed at different installation locations within the sensing range, An information processing device, A sensing system including Each of the sensor nodes includes a sensor unit that detects sensor information at the installation location of the sensor node, an irradiation unit that irradiates visible light, and a control unit that controls the irradiation unit to irradiate visible light according to the sensor information. The moving body sequentially moves to each of the sensor nodes, performs visible light communication for acquiring the sensor information by capturing the visible light irradiated by the irradiation unit of each of the sensor nodes with the camera, The information processing device Based on the transmission rate of the visible light communication and the maximum data size of the sensor information, obtains the maximum transmission time required for the visible light communication of the sensor information, Outputs an operation command so that the moving body captures the visible light with the camera for a time equal to or longer than the maximum transmission time at the sensor node. Sensing system. [Appendix 2] The sensing system according to Appendix 1, wherein the control unit controls the irradiation unit to irradiate visible light according to the sensor information when the sensor unit detects the moving body at the installation location of the sensor node. [Appendix 3] The sensing system according to Appendix 2, wherein the sensor unit detects the moving body by detecting infrared rays emitted by the moving body. [Appendix 4] The sensing system according to any one of Appendices 1 to 3, wherein the control units of the plurality of sensor nodes control the irradiation unit to irradiate a visible light signal including identification information unique to each of the sensor nodes. [Appendix 5] The sensing system according to Appendix 4, wherein the moving body identifies the sensor node based on the visible light signal photographed by the camera. [Appendix 6] The sensing system according to Appendix 4, wherein the control unit controls the irradiation unit to irradiate visible light according to the sensor information continuously following the identification information. [Appendix 7] The information processing device obtains, for each of the plurality of sensor nodes, the maximum transmission time required for visible light communication of the sensor information based on the transmission rate of the visible light communication and the maximum data size of the sensor information of the sensor node, The moving body outputs a driving command so that the camera photographs the visible light for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes. The sensing system according to any one of Appendices 1 to 6. [Appendix 8] The information processing device further clusters the plurality of sensor nodes based on the installation locations of the sensor nodes, The sensing system according to Appendix 7, wherein the moving route of the moving body is determined based on a graph including virtual sensor nodes representing each cluster obtained by the clustering and edges connecting the virtual sensor nodes. [Appendix 9] The moving body is a sensing system according to any one of Appendices 1 to 7, which is an aircraft. [Appendix 10] The sensing range is a farm, The sensor information is a sensing system according to any one of Appendices 1 to 9, including temperature, humidity, illuminance, or soil moisture. [Appendix 11] The sensing range is underwater, The moving body is a sensing system according to any one of Appendices 1 to 9, which is a submersible. [Appendix 12] The sensing range is indoors, The moving body is a sensing system according to any one of Appendices 1 to 9, which is a self-propelled moving body. [Appendix 13] A sensing system according to any one of Appendices 1 to 12, which estimates the position of the moving body from the position of the sensor node determined in advance and the positional relationship of the sensor node on the image captured by the camera. [Appendix 14] A moving body equipped with a camera, A plurality of sensor nodes provided at different installation locations within the sensing range, An information processing device, A sensing method in a sensing system including: Each sensor unit of the sensor node detects sensor information at the installation location of the sensor node, and a control unit controls an irradiation unit that irradiates visible light so as to irradiate visible light according to the sensor information. The moving body sequentially moves to each of the sensor nodes, captures visible light irradiated by the irradiation unit of each of the sensor nodes with the camera, and performs visible light communication to acquire the sensor information. The information processing device Obtains the maximum transmission time required for visible light communication of the sensor information based on the transmission rate of the visible light communication and the maximum data size of the sensor information. The mobile body outputs an operation command so as to photograph the visible light with the camera for a time longer than the maximum transmission time at the sensor node. Sensing method.

[0112] The disclosure of Japanese Application No. 2022-107368 is incorporated herein by reference in its entirety.

[0113] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A mobile body equipped with a camera, a plurality of sensor nodes installed at different installation locations within the sensing range, an information processing device, and a sensing system including: Each of the sensor nodes includes a sensor unit that detects sensor information at the installation location of the sensor node, an irradiation unit that irradiates visible light, and a control unit that controls the irradiation unit to irradiate visible light according to the sensor information. The mobile body sequentially moves to each of the sensor nodes, performs visible light communication to acquire the sensor information by photographing the visible light irradiated by the irradiation unit of each of the sensor nodes with the camera. The information processing device For each of the plurality of sensor nodes, based on the transmission rate of the visible light communication and the maximum data size of the sensor information, obtains the maximum transmission time required for the visible light communication of the sensor information of the sensor node. Outputs an operation command so that the mobile body photographs the visible light with the camera for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes. A sensing system.

2. The sensing system according to claim 1, wherein the control unit controls the irradiation unit to irradiate visible light according to the sensor information when the sensor unit detects the mobile body at the installation location of the sensor node.

3. The sensing system according to claim 2, wherein the sensor unit detects the mobile body by detecting infrared rays emitted by the mobile body.

4. The sensing system according to claim 1, wherein the control units of the plurality of sensor nodes control the irradiation unit to irradiate a visible light signal including unique identification information for each of the sensor nodes.

5. The sensing system according to claim 4, wherein the mobile body identifies the sensor node based on the visible light signal photographed by the camera.

6. The sensing system according to claim 4, wherein the control unit controls the irradiation unit to irradiate visible light according to the sensor information continuously following the identification information.

7. The information processing device further Clusters the plurality of sensor nodes based on the installation locations of the sensor nodes, Determines the movement route of the mobile body based on a graph including virtual sensor nodes representing each cluster obtained by the clustering and edges connecting the virtual sensor nodes. The sensing system according to claim 1.

8. The sensing system according to claim 1, wherein the moving body is an aircraft.

9. The sensing range is an agricultural garden, The sensing system according to claim 1, wherein the sensor information includes temperature, humidity, illuminance, or soil moisture.

10. The sensing range is underwater, The sensing system according to claim 1, wherein the moving body is a submersible.

11. The sensing range is indoors, The sensing system according to claim 1, wherein the moving body is a self-propelled moving body.

12. The sensing system according to claim 1, wherein the position of the moving body is estimated from the position of the sensor node determined in advance and the positional relationship of the sensor node on the image captured by the camera.

13. A moving body equipped with a camera, A plurality of sensor nodes provided at different installation locations within the sensing range, An information processing device, A sensing method in a sensing system including: Each sensor unit of each of the sensor nodes detects sensor information at the installation location of the sensor node, and a control unit controls an irradiation unit that irradiates visible light so as to irradiate visible light according to the sensor information. The moving body sequentially moves to each of the sensor nodes, performs visible light communication for acquiring the sensor information by capturing the visible light irradiated by the irradiation unit of each of the sensor nodes with the camera. The information processing device For each of the plurality of sensor nodes, obtains the maximum transmission time required for visible light communication of the sensor information of the sensor node based on the transmission rate of the visible light communication and the maximum data size of the sensor information. Outputs an operation command so that the moving body captures the visible light with the camera for a time equal to or longer than the maximum transmission time at each of the plurality of sensor nodes. Sensing method.

Images