Receiving device and optical wireless communication system

Abstract

To provide a technique for reducing the number of light sources required to estimate the position and orientation of a camera by optical wireless communication.SOLUTION: A receiving device includes a camera and an information processing device that receives, via the camera, an optical wireless communication signal transmitted from each of multiple light sources. The optical wireless communication signal includes position related information for specifying the position in an absolute coordinate system of a transmission source of the optical wireless communication signal. The information processing device performs camera position estimation. The camera position estimation estimates the position and orientation of the camera in the absolute coordinate system based on a light source image position and a light source absolute position. A first light source included in the multiple light sources transmits a first optical wireless communication signal. The first optical wireless communication signal includes first position related information for specifying positions of multiple feature points included in the first light source. The information processing device is configured to perform camera position estimation based on the first position related information, by regarding each of the multiple feature points as part of the multiple light sources.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to optical wireless communication technology using a camera. [Background technology]

[0002] Patent Document 1 discloses an optical marker system that can stably estimate the position and orientation of a moving imaging device. The imaging device (camera) captures an image including the light-emitting points and characteristic points of the LED marker. The LED marker is identified based on the blinking pattern detected from the captured image, and the position and orientation of the camera are estimated from the three-dimensional position, etc., of the identified LED marker. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-033366 Summary of the Invention [Problem to be solved by the invention]

[0004] Estimating the position and orientation of a camera requires position information about multiple light sources. In the technology disclosed in Patent Document 1, one light source (LED marker) transmits only its own position information. This means that the number of light sources required to estimate the position and orientation of a camera tends to be large. As a related issue, there is a high possibility that the required number of light sources will not be captured within the camera's angle of view.

[0005] One object of the present disclosure is to provide a technique for reducing the number of light sources required to estimate the position and orientation of a camera through optical wireless communication. [Means for solving the problem]

[0006] The first aspect relates to a receiving device. The receiving device includes a camera and an information processing device that receives, via the camera, the optical wireless communication signals transmitted from each of the plurality of light sources. The optical wireless communication signal includes position-related information for identifying the position of the source of the optical wireless communication signal in an absolute coordinate system. The information processing device executes camera position estimation. The camera position estimation is Identifying a signal region of the optical wireless communication signal in an image plane acquired by the camera; obtaining light source image positions indicating the projected positions of each of a plurality of light sources onto an image plane; acquiring light source absolute positions indicating positions of each of the plurality of light sources in an absolute coordinate system based on position-related information included in the optical wireless communication signal; estimating a position and orientation in an absolute coordinate system of the camera based on the light source image position and the light source absolute position; Includes: A first light source included in the plurality of light sources transmits a first optical wireless communication signal. The first wireless optical communication signal includes first position-related information for identifying positions of a plurality of characteristic points included in the first light source. The information processing device is configured to perform camera position estimation by regarding each of the plurality of feature points as a part of the plurality of light sources based on the first position-related information.

[0007] The second aspect relates to an optical wireless communication system. Optical wireless communication systems include: a receiving device equipped with a camera; an information processing device that receives, via a camera, optical wireless communication signals transmitted from each of the plurality of light sources; Equipped with. The optical wireless communication signal includes position-related information for identifying the position of the source of the optical wireless communication signal in an absolute coordinate system. The information processing device executes camera position estimation. The camera position estimation is Identifying a signal region of the optical wireless communication signal in an image plane acquired by the camera; obtaining light source image positions indicating the projected positions of each of a plurality of light sources onto an image plane; acquiring light source absolute positions indicating positions of each of the plurality of light sources in an absolute coordinate system based on position-related information included in the optical wireless communication signal; estimating a position and orientation in an absolute coordinate system of the camera based on the light source image position and the light source absolute position; Includes: A first light source included in the plurality of light sources transmits a first optical wireless communication signal. The first wireless optical communication signal includes first position-related information for identifying positions of a plurality of characteristic points included in the first light source. The information processing device is configured to perform camera position estimation by regarding each of the plurality of feature points as a part of the plurality of light sources based on the first position-related information. [Effects of the Invention]

[0008] According to the first and second aspects, the information processing device performs camera position estimation by regarding each of a plurality of feature points included in the first light source as part of a plurality of light sources, thereby contributing to reducing the number of light sources required for camera position estimation. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an overview of an optical wireless communication system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a perspective view showing the geometric relationship between the light source absolute position and the light source image position. [Figure 3] FIG. 10 is a schematic diagram showing the process of acquiring the light source image position. [Figure 4] 1 is a graph showing data frequency characteristics of an optical signal and noise. [Figure 5] 1 is a schematic diagram illustrating some examples of light source absolute position acquisition; [Figure 6] 10A and 10B are diagrams illustrating an image plane before and after light source image position acquisition and light source absolute position acquisition are performed. [Figure 7] FIG. 1 is a schematic diagram of applied camera position estimation. [Figure 8] FIG. 10 is a block diagram showing an outline of acquisition of the absolute positions of the characteristic points of the first light source. [Figure 9] 10A to 10C are schematic diagrams showing some examples of image position acquisition of feature points. [Figure 10] 1A and 1B are schematic diagrams illustrating some examples of determining the shape of a signal region using variance. [Figure 11] FIG. 1 is a block diagram showing a first configuration example of an optical wireless communication system. [Figure 12] FIG. 10 is a block diagram showing a second configuration example of the optical wireless communication system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0011] 1.Basic configuration 1 is a diagram illustrating an overview of an optical wireless communication system 1 according to the present embodiment. The optical wireless communication system 1 includes a plurality of light sources 10-1 to 10-n (n is an integer satisfying n≧2), a receiving device 20 equipped with an event camera 21, and an information processing device 30.

[0012] The multiple light sources 10-1 to 10-n are fixedly attached in space and installed indoors or outdoors. Examples of the multiple light sources 10-1 to 10-n include visible light LEDs (light emitting diodes) and infrared LEDs. Visible light LEDs are widely used in street lights, indoor lights, traffic lights, electronic billboards, etc., so using visible light LEDs in the optical wireless communication system 1 leads to effective use of existing facilities. Visible light LEDs repeatedly blink at a speed that is imperceptible to the human eye, so by controlling the blinking, visible light can be used as a communication signal. It is preferable that the multiple light sources 10-1 to 10-n are simultaneously included in the angle of view of the event camera 21.

[0013] A light source 10-i (i = 1 to n) constituting the plurality of light sources 10-1 to 10-n transmits an optical wireless communication signal Si by blinking. The optical wireless communication signal Si includes position related information Pi for specifying an absolute light source position AP-i indicating the position of the light source 10-i, which is the source of the signal, in an absolute coordinate system. In the drawings of the present disclosure, the absolute coordinate system is represented by the X axis, the Y axis, and the Z axis. The absolute light source position AP-i of the light source 10-i is also expressed by [X i ,Y i ,Z i Hereinafter, for simplicity, the "optical wireless communication signal Si" will be simply referred to as the "optical signal Si."

[0014] The receiving device 20 includes an event camera 21. Preferably, the receiving device 20 simultaneously receives the optical signals S-1 to S-n transmitted from each of the multiple light sources 10-1 to S-n. Typical examples of the receiving device 20 include mobile devices such as smartphones and tablets, as well as augmented reality devices. Examples of augmented reality devices include augmented reality goggles or glasses worn on the face. The display of the augmented reality device displays real images and digital information (text, 3D avatars, other images, etc.) superimposed on each other. Furthermore, the receiving device 20 is not limited to these examples, and any object equipped with the event camera 21 can function as the receiving device 20. For example, if a vehicle, a robot, a wheelchair, a cane, or the like is equipped with the event camera 21, it can function as the receiving device 20. In this way, the receiving device 20 is typically an object or terminal that is not fixed in space. When the receiving device 20 moves, the event camera 21 also moves accordingly.

[0015] The event camera 21 has a built-in event-based vision sensor (EV sensor). The EV sensor observes changes in the luminance of light received by the pixels (image sensors) within the EV sensor. When the EV sensor observes a luminance change equal to or greater than a preset threshold, it detects the luminance change as an "event." An event is detected when a situation different from the previous situation occurs. For example, when the subject or the event camera 21 moves, the relative position between the subject and the event camera 21 changes, causing the subject to appear in a pixel where it had not previously existed. At this time, a significant luminance change occurs in the pixels surrounding the subject, which is detected as an event. In addition, the optical signal Si also generates a luminance change due to the blinking of the light source 10-i, and is therefore detected as an event.

[0016] The event camera 21 outputs data related to a pixel where an event is detected as event data EVD. The event data EVD includes at least the coordinates on the image plane of the pixel where the event occurred, the time when the event was detected, and the brightness polarity (positive / negative). The threshold value referenced by the EV sensor when detecting a brightness change is set to be changeable. When an event occurs, threshold values ​​are set on the positive side (a change toward brightness) and the negative side (a change toward darkness) of a voltage (reference voltage) based on the brightness level at that time. A voltage change exceeding the positive threshold is detected as a positive event, and a voltage change exceeding the negative threshold is detected as a negative event. In other words, the EV sensor detects the light signal Si when the light source 10-i turns on as a positive event, and when the light source 10-i turns off as a negative event.

[0017] In this embodiment, an event camera 21 is used as the imaging device (camera), but a normal camera (frame-based camera) may also be used. A frame-based camera outputs all received light as an image, whereas the event camera 21, as described above, outputs only pixels where an event is detected. Therefore, the amount of data handled by the event camera 21 is smaller than that of a normal camera, and as a result, the time resolution is higher than that of a normal camera. Since the event camera 21 is more suitable for high-speed communication than a normal camera, the event camera 21 is suitable in situations where high-speed communication such as visible light communication is required. On the other hand, a normal camera may be used in situations where high-speed communication is not required.

[0018] The information processing device 30 acquires event data EVD from the event camera 21. The information processing device 30 can recognize the optical signal Si by extracting a component corresponding to the optical signal Si from the event data EVD. That is, the information processing device 30 can receive the optical signal Si through the event camera 21. Furthermore, the information processing device 30 acquires position-related information Pi by demodulating the received optical signal Si. The information processing device 30 acquires the two-dimensional position of the optical signal Si on the image plane using the event data EVD. This two-dimensional position indicates the position where the light source 10-i is projected on the image plane. Hereinafter, the position where the light source 10-i is projected on the image plane will be referred to as the "light source image position IP-i," and the process of acquiring the light source image position IP-i will be referred to as "light source image position acquisition." In the present disclosure, the image plane coordinate system is represented by the u axis and the v axis. Furthermore, the light source image position IP-i of the light source 10-i can be expressed as [u i ,v i The information processing device 30 may be included in the receiving device 20 or may be an external device to the receiving device 20.

[0019] Furthermore, the information processing device 30 acquires the three-dimensional position (light source absolute position AP-i) of the light source 10-i in the absolute coordinate system from the position related information Pi included in the optical signal Si. Hereinafter, the process of acquiring the light source absolute position AP-i will be referred to as "light source absolute position acquisition." Specific examples of the position related information Pi and the light source absolute position acquisition will be described later.

[0020] Through the above-described steps, the information processing device 30 calculates the light source absolute position AP-i ([X i ,Y i ,Z i ]) and the light source image position IP-i ([u i ,v i ]). FIG. 2 is a perspective view showing the geometric relationship between the light source absolute position AP-i and the light source image position IP-i. The information processing device 30 estimates the position and orientation of the event camera 21 in the absolute coordinate system from the geometric relationship between the n sets of data sets. More specifically, a rotation matrix and a translation vector are obtained from the n sets of data sets. A specific solution is known as the PnP (Perspective n Point) problem, and the number of data sets (i.e., the value of n) required to solve this problem varies depending on the method. For example, a method called the 8-point algorithm is known as one solution when n = 8. Note that since the event camera 21 is provided in the receiving device 20, estimating the absolute position and orientation of the event camera 21 is synonymous with estimating the absolute position and orientation of the receiving device 20.

[0021] As described above, the information processing device 30 estimates the absolute position and orientation of the event camera 21 by acquiring the light source image position and the light source absolute position. This series of processes is hereinafter referred to as "camera position estimation." Existing position estimation systems (e.g., satellite positioning systems) may not be able to accurately determine position in places where satellite radio waves are difficult to reach (inside buildings, underground, between high-rise buildings, etc.). On the other hand, camera position estimation using the optical wireless communication system 1 does not use satellite radio waves, so it can be said that there are fewer restrictions on the places where it can be used.

[0022] 2. Camera position estimation A series of processes relating to camera position estimation will be described in detail below.

[0023] 2-1.Light source image position acquisition FIG. 3 is a schematic diagram showing the process of acquiring the light source image position.

[0024] 3A is a graph showing the spatiotemporal distribution of the event data EVD received by the information processing device 30 from the event camera 21. The event data EVD includes not only events caused by the blinking of the light source 10-i (i.e., the optical signal Si) but also noise N, which is an event caused by the movement of the subject or the event camera 21. In other words, the event data EVD is output in a state where the optical signal Si necessary for camera position estimation and the noise N unnecessary for camera position estimation are mixed together. Therefore, in order for the information processing device 30 to acquire the light source image position IP-i, it is necessary to perform signal separation, which is a process of separating the optical signal Si from the noise N.

[0025] One method of signal separation is based on the "data frequency" of the event data EVD for each pixel. As described above, the event data EVD includes the time at which the event was detected. Therefore, the information processing device 30 can calculate the number of event data EVD detected per unit time for each pixel. The number of event data EVD detected per unit time can be called the "data frequency."

[0026] FIG. 4 is a graph showing the data frequency characteristics of the optical signal Si and the noise N. The vertical axis represents the number of pixels detected by the EV sensor. The horizontal axis represents the data frequency. As can be seen from the graph, the data frequency characteristics of the optical signal Si and the noise N are significantly different. The data frequency of the optical signal Si is linked to the flickering frequency of the light source 10-i, and its value is approximately several hundred Hz to several hundred kHz. On the other hand, the data frequency of the noise N caused by the movement of the subject or the event camera 21 is significantly smaller (approximately several tens of Hz) than the data frequency of the optical signal Si. Therefore, the information processing device 30 can separate the optical signal Si from the noise N by using a frequency filter. An example of a frequency filter is a high-pass filter that cuts off signals below a predetermined frequency. In this case, the information processing device 30 determines that a pixel region in which a high data frequency not cut off by the high-pass filter is observed is a signal region occupied by the optical signal Si.

[0027] (B) in FIG. 3 is a graph showing the spatiotemporal distribution of the event data EVD after signal separation has been performed. As described above, the event data EVD includes information on coordinates in the image plane coordinate system, so the information processing device 30 can acquire the position of the separated optical signal Si in the image plane coordinate system. The position of the optical signal Si in the image plane indicates the position where the light source 10-i is projected onto the image plane, i.e., the light source image position IP-i. In reality, the signal area occupied by the optical signal Si on the image plane extends over multiple pixels, so for example, the center coordinate of the signal area of ​​the optical signal Si may be considered to be the light source image position IP-i. In this manner, the light source image position acquisition is performed.

[0028] 2-2. Obtaining the absolute position of the light source 5 is a schematic diagram showing some examples of acquiring the light source absolute position. As described above, the optical signal Si transmitted from the light source 10-i includes position-related information Pi for identifying the light source absolute position AP-i.

[0029] 5A, the optical signal Si transmitted from the light source 10-i includes the light source absolute position AP-i as position-related information Pi. The receiving device 20 receives the optical signal Si via the event camera 21. The receiving device 20 passes the event data EVD output by the event camera 21 to the information processing device 30. The event data EVD records the content of the optical signal Si as a change in brightness, so the information processing device 30 obtains the position-related information Pi, i.e., the light source absolute position AP-i, based on the event data EVD.

[0030] 5B, the optical signal Si includes identification information SID-i, which is information for identifying the light source 10-i. In this case, the optical wireless communication system 1 further includes a storage device 50. The storage device 50 stores absolute light source positions AP-1 to AP-n of the multiple light sources 10-1 to AP-n in association with the identification information SID-1 to AP-n of the multiple light sources 10-1 to AP-n. The information processing device 30 accesses the storage device 50 and acquires the absolute light source position AP-i corresponding to the identification information SID-i. The storage device 50 may be built into the information processing device 30 or may be an external device different from the information processing device 30. Alternatively, the storage device 50 may be managed by a management server, and the information processing device 30 may acquire the absolute light source position AP-i through communication with the management server.

[0031] 6A and 6B are diagrams showing the image plane before and after the light source image position acquisition and the light source absolute position acquisition are performed. (A) in Fig. 6A shows the state of the image plane before the light source image position acquisition and the light source absolute position acquisition are performed, i.e., at the time when the information processing device 30 acquires the event data EVD. At this time, the event data EVD includes noise N as well as the optical signal Si.

[0032] The information processing device 30 separates the optical signal Si from the noise N and acquires the position of the optical signal Si in the image plane coordinate system, i.e., the light source image position IP-i of the light source 10-i. The information processing device 30 also acquires the light source absolute position AP-i of the light source 10-i through light source absolute position acquisition. As a result of the light source image position acquisition and light source absolute position acquisition, the information processing device 30 acquires a data set of the light source absolute position AP-i and the light source image position IP-i relative to the light source 10-i, as shown in (B) of FIG. 6. The information processing device 30 can estimate the position and orientation of the event camera 21 in the absolute coordinate system from these data sets and the geometric relationship based on the focal length of the event camera 21.

[0033] 3. Applied Camera Position Estimation Camera position estimation requires a data set including light source absolute positions AP-1 to n and light source image positions IP-1 to n for each of the multiple light sources 10-1 to n. However, the multiple light sources 10-1 to n do not necessarily fall within the angle of view of the event camera 21. Furthermore, if multiple light sources 10-1 to n are always to be accommodated within the angle of view of the event camera 21, additional light sources must be installed to make it easier for n or more light sources to fall within the angle of view of the event camera 21. Therefore, this embodiment focuses on the fact that the actual light source 10-i is not a point, but a unit having a linear or planar range.

[0034] Consider a case where multiple feature points are set within the range of the first light source 10-1. If camera position estimation can be performed by regarding each of these multiple feature points as a different light source 10-i (i.e., part of the multiple light sources 10-1 to 10-n), the number of installed light sources can be reduced. The camera position estimation performed by the information processing device 30 by regarding each of the multiple feature points as a different light source 10-i is particularly referred to as "advanced camera position estimation."

[0035] FIG. 7 is a schematic diagram of applied camera position estimation.

[0036] A first light source 10-1 attached to a pole has a plurality of feature points. In FIG. 7, the first light source 10-1 includes feature points CP1 and CP2. A first wireless optical communication signal S-1 (hereinafter referred to as the first optical signal S-1) transmitted by the first light source 10-1 includes first position-related information P-1. The first position-related information P-1 is information for identifying the absolute position (feature point absolute position) of each feature point of the first light source 10-1. Upon receiving the first optical signal S-1, the information processing device 30 executes a process of acquiring the feature point absolute positions based on the first position-related information P-1 (hereinafter referred to as "acquiring the absolute position of the feature point"). In FIG. 7, the information processing device 30 acquires an absolute feature point position APc1 (Xc1, Yc1, Zc1) corresponding to the feature point CP1 and an absolute feature point position APc2 (Xc2, Yc2, Zc2) corresponding to the feature point CP2.

[0037] The information processing device 30 executes a process of acquiring feature point image positions indicating the positions where each feature point of the first light source 10-1 is projected onto the image plane (hereinafter referred to as "acquiring image positions of feature points"). In Fig. 7, the information processing device 30 acquires feature point image positions IPc1 (uc1, vc1) and feature point image positions IPc2 (uc2, vc2).

[0038] The information processing device 30 executes a process of associating each feature point absolute position with a feature point image position (feature point association). In Fig. 7, the feature point absolute position APc1 is associated with the feature point image position IPc1, and the feature point absolute position APc2 is associated with the feature point image position IPc2.

[0039] Through the above-described series of processes, the information processing device 30 acquires a feature point data set consisting of the absolute feature point position and the corresponding feature point image position for each feature point included in the first light source 10-1. That is, multiple feature point data sets are acquired from the first light source 10-1. This allows the information processing device 30 to estimate the absolute position and orientation of the event camera 21 by regarding each feature point included in the first light source 10-1 as an independent light source 10-i.

[0040] In this way, the information processing device 30 executes the advanced camera position estimation. Each process related to the advanced camera position estimation will be described in detail below.

[0041] 3-1. Obtaining the absolute position of feature points 8 is a block diagram showing an outline of how the absolute positions of the characteristic points of the first light source 10-1 are acquired. The basic components are the same as those in FIG.

[0042] In the case of (A) in FIG. 8, the first position-related information P-1 includes feature point position information APc. The feature point position information APc includes information about the absolute position of each feature point of the first light source 10-1, such as the feature point absolute positions APc1 and APc2 shown in FIG. 7. The information processing device 30 acquires the first position-related information P-1, i.e., the feature point position information APc, based on the event data EVD. In the case of (B) in FIG. 8, the feature point position information APc is stored in the storage device 50. The information processing device 30 acquires the feature point position information APc in the storage device 50 via the first identification information SID-1 included in the first optical signal S-1, similar to (B) in FIG. 5. In this way, the information processing device 30 acquires the absolute position of each feature point included in the first light source 10-1.

[0043] The feature point position information APc is set in advance. The feature point position information APc is set according to the shape of the first light source 10-1. For example, if the first light source 10-1 has an elongated linear shape (linear), two points on both ends of the first light source 10-1 are set as feature points. If the first light source 10-1 has a circular shape, a total of two points, the center point of the first light source 10-1 and one point on the circumference, are set as feature points.

[0044] 3-2. Obtaining image positions of feature points The image positions of the feature points are acquired in a manner that corresponds to the shape of the first light source 10-1. Figure 9 is a schematic diagram showing some examples of acquiring the image positions of the feature points.

[0045] 9A is an example showing how to obtain image positions of feature points when the first light source 10-1 is linear. In this case, the information processing device 30 identifies the points (pixels) at the "both ends" of the signal area as image feature points. The same number of image feature points as the number of feature points included in the first light source 10-1 are identified. The two ends of the signal area are found by extracting the combination of two points included in the signal area that is the furthest apart.

[0046] (B) in FIG. 9 is an example showing how to acquire the image position of a feature point when the first light source 10-1 is circular. The information processing device 30 identifies the "center point" and "one point on the circumference" of the signal area as image feature points. The center point is found as the intersection of lines that bisect the range of the signal area along each of two arbitrary orthogonal axes. The center point can also be considered the center of gravity of the data that forms the signal area. The one point on the circumference is found as the point within the signal area that is farthest from the center point.

[0047] The information processing device 30 acquires the positions of image feature points on the image plane (feature point image positions). In (A) of Fig. 9, the information processing device 30 acquires a feature point image position IPc1 (one end) and a feature point image position IPc2 (the other end). In (B) of Fig. 9, the information processing device 30 acquires a feature point image position IPc1 (center point) and a feature point image position IPc2 (point on the circumference).

[0048] 3-3. Feature point matching After acquiring the absolute positions and image positions of the feature points, the information processing device 30 acquires the feature point absolute positions and feature point image positions of the first light source 10-1. However, it is not obvious which feature point absolute positions correspond to which feature point image positions. Referring to FIG. 7, each of the feature point image positions IPc1 and IPc2 is the position where either the feature point absolute position APc1 or APc2 is projected onto the image plane, but the correspondence between them is not uniquely determined. This correspondence problem arises because the first position-related information P-1 is information indicating multiple feature points included in the first light source 10-1. Note that this problem does not occur in the camera position estimation (basic camera position estimation) described in Sections 1 and 2. This is because in basic camera position estimation, the position-related information Pi is information indicating the absolute position (one coordinate) of the light source 10-i that is the transmission source.

[0049] The problem of correspondence between the absolute positions of feature points and the image positions of feature points can be solved by using light sources 10-i other than the first light source 10-1. With only the first light source 10-1, there is a degree of freedom between the absolute positions of feature points and their positions on the image, but the presence of other light sources 10-i increases the constraint conditions (i.e., reduces the degrees of freedom), and the correspondence between the absolute positions of feature points and the image positions of feature points is determined uniquely and without contradiction.

[0050] Through the above processing, the information processing device 30 acquires multiple corresponding feature point data sets from the first light source 10-1. The information processing device 30 can estimate the absolute position and orientation of the event camera 21 using the corresponding feature point data sets. In other words, the applied camera position estimation is a camera position estimation that is performed by regarding each of the multiple feature points of the first light source 10-1 as an independent light source 10-i (part of the multiple light sources 10-1 to 10-n).

[0051] 3-4.Light source shape identification Consider a case where light sources of different shapes are used simultaneously in the optical wireless communication system 1. In this case, the information processing device 30 identifies the shape of the first light source 10-1 before starting to acquire the image positions of the feature points. This process is hereinafter referred to as light source shape identification. As described above, the image positions of the feature points are acquired in a manner corresponding to the shape of the first light source 10-1, so light source shape identification is necessary to acquire the image positions of the feature points in an appropriate manner. Note that if the shapes of the light sources used in the optical wireless communication system 1 are uniform, the process for acquiring the image positions of the feature points does not change depending on the shape of the first light source 10-1, and so light source shape identification is not necessary. Several examples of light source shape identification are shown below.

[0052] <First example> A first example of light source shape acquisition is a case where the first light source 10-1 transmits shape identification information SHP, which is information for identifying its own shape (e.g., linear, circular, etc.). The first example will be described with reference to FIG. 8. In (A) of FIG. 8, the shape identification information SHP is included in the first optical signal S-1 transmitted by the first light source 10-1. That is, the information processing device 30 can acquire the shape of the first light source 10-1 by referring to the shape identification information SHP included in the first optical signal S-1. In (B) of FIG. 8, the shape identification information SHP is stored in the storage device 50. The information processing device 30 acquires the shape identification information SHP in the storage device 50 via the first identification information SID-1 included in the first optical signal S-1.

[0053] <Second example> It is also conceivable that the information processing device 30 determines the shape of the signal region of the first optical signal S-1 and performs light source shape identification based on the result. One example is a method that utilizes the variance of the signal region on the image plane. FIG. 10 is a schematic diagram showing several examples of signal region shape determination using variance. The information processing device 30 sets a first axis (e1 axis in FIG. 10) that passes through the center of gravity of the signal region and maximizes the variance (variation) of data from the center of gravity. The information processing device 30 sets a second axis (e2 axis in FIG. 10) that passes through the center of gravity of the signal region and is perpendicular to the first axis. The information processing device 30 calculates the variance V1 in the first axis direction, the variance V2 in the second axis direction, and the variance ratio (V1 / V2).

[0054] (A) in Figure 10 shows a case where the dispersion ratio is relatively large. In other words, it shows that the variation in the first axis direction is significantly larger than the variation in the second axis direction. This shows that the shape of the signal region is elongated along the first axis, i.e., linear. From this result, the information processing device 30 determines that the shape of the first light source 10-1 is also linear.

[0055] (B) in Figure 10 shows a case where the dispersion ratio is relatively small. In other words, it shows that there is no significant difference between the variation in the first axis direction and the variation in the second axis direction. This indicates that the shape of the signal region varies equally along both axes. For example, assuming that the shape of the first light source 10-1 is either linear or circular, the information processing device 30 can determine that the first light source 10-1 has a non-linear shape, i.e., a circular shape.

[0056] In this way, the information processing device 30 can determine the shape of the first light source 10-1. More specifically, the shape of the first light source 10-1 is determined based on the magnitude relationship between the threshold values ​​for determining each shape and the dispersion ratio.

[0057] Effects As described above, the information processing device 30 performs camera position estimation by regarding each of the multiple feature points included in the first light source 10-1 as a different light source 10-i, that is, as part of the multiple light sources 10-1 to 10-n, thereby reducing the number of installed light sources.

[0058] The information processing device 30 can also acquire the shape of the first light source 10-1. This allows the information processing device 30 to appropriately perform the applied camera estimation process even when light sources of different shapes are used simultaneously in the optical wireless communication system 1.

[0059] As described above, the imaging device (camera) included in the receiving device 20 may be a normal camera instead of the event camera 21. The information processing device 30 can also perform camera position estimation based on the light source detection result of the normal camera. Even in this case, the effect of reducing the number of installed light sources can be similarly obtained.

[0060] 4.Configuration example FIG. 11 is a block diagram showing a first configuration example of the optical wireless communication system 1. As shown in FIG.

[0061] The blinking control device 60 controls the blinking pattern of the light source 10-i. The blinking control device 60 may be built into each facility (street light, indoor light, etc.) that includes the light source 10-i. Alternatively, the blinking control device 60 may be included in an external facility (such as a management server) and control the blinking of the light source 10-i from outside. Furthermore, when the blinking control device 60 is provided in an external facility, it may collectively control the blinking of multiple light sources 10-1 to 10-n.

[0062] The information generating unit 61 generates a digital signal D. The digital signal D is a signal that represents the position-related information Pi using two values, “0” and “1.” The generated digital signal D is output to the modulation unit 62.

[0063] The modulator 62 generates a modulated signal M by modulating the digital signal D into a signal suitable for optical wireless communication. As a modulation method, a pulse width modulation (PWM) method, a pulse position modulation (PPM) method, or the like is used. The PWM method is a method in which the ratio of the ON (on) and OFF (off) time of the light source 10-i is changed by an input signal. The PPM method is a modulation method in which the position of the carrier pulse on the time axis is changed by an input signal.

[0064] The blinking control device 60 passes the modulation signal M to the light source 10-i. The light source 10-i blinks in accordance with the modulation signal M. The optical signal Si is a signal indicated by the blinking pattern of the light source 10-i expressed by the modulation signal M.

[0065] The receiving device 20 receives the optical signal Si through the event camera 21. Specifically, an EV sensor built into the event camera 21 detects a change in brightness due to the optical signal Si as an event. The receiving device 20 transmits the event data EVD to a signal separation unit 31 in the information processing device 30.

[0066] 11, the information processing device 30 is a device (e.g., an external server) different from the receiving device 20. That is, the receiving device 20 transmits event data EVD to the external information processing device 30. The information processing device 30 executes a series of subsequent processes based on the received event data EVD.

[0067] The signal separation unit 31 acquires the light source image position IP-i by performing the process (the signal separation described above) of separating the optical signal Si and the noise N included in the event data EVD. The light source image position IP-i is output to the position estimation unit 33.

[0068] The demodulator 32 demodulates the optical signal Si separated by the signal separator 31 to obtain the position-related information Pi. The demodulator 32 passes the position-related information Pi to the position estimator 33.

[0069] The position estimation unit 33 acquires the light source absolute position AP-i based on the position relation information Pi. For example, as in (A) of FIG. 5, when the light source absolute position AP-i is directly transmitted as the position relation information Pi, the position estimation unit 33 directly acquires the light source absolute position AP-i. Also, as in (B) of FIG. 5, when the identification information SID-i is transmitted as the position relation information Pi, the information processing device 30 accesses the storage device 50 and acquires the light source absolute position AP-i corresponding to the identification information SID-i. The information processing device 30 acquires n sets of data sets of the light source absolute position AP-i and the light source image position IP-i relative to the light source 10-i. The information processing device 30 estimates the position and orientation of the event camera 21 in the absolute coordinate system from the geometric relationship between these n sets of data sets.

[0070] Fig. 12 is a block diagram showing a second configuration example of the optical wireless communication system 1. The basic configuration is similar to the example in Fig. 11. In the example in Fig. 12, the information processing device 30 is built into the receiving device 20. In this case, after receiving the optical signal Si, the receiving device 20 can complete camera position estimation within the receiving device 20. [Explanation of symbols]

[0071] 10-1: 1st light source 10-i: Light source 20: Receiving device 21: Event Camera 30: Information processing device 50: Storage device AP-i: Light source absolute position APc: feature point location information EVD: Event Data IP-1: Light source image position IP-i: Light source image position IPc1: Feature point image position IPc2: Feature point image position N: Noise P-1: First location-related information Pi: Location-related information S-1: First optical wireless communication signal Si: Optical wireless communication signal SHP: Shape specific information SID-1: First identification information SID-i: Identification information

Claims

1. A camera and an information processing device that receives, via the camera, optical wireless communication signals transmitted from each of a plurality of light sources; Equipped with The optical wireless communication signal and position-related information for identifying a position in an absolute coordinate system of a source of the optical wireless communication signal, The information processing device performs camera position estimation; The camera position estimation is identifying a signal region of the optical wireless communication signal in an image plane acquired by the camera; obtaining a light source image position indicating a position where each of the plurality of light sources is projected onto the image plane; acquiring a light source absolute position indicating a position of each of the plurality of light sources in the absolute coordinate system based on the position related information included in the optical wireless communication signal; estimating a position and orientation of the camera in the absolute coordinate system based on the light source image position and the light source absolute position; Including, a first light source included in the plurality of light sources transmits a first optical wireless communication signal; the first wireless optical communication signal includes first position-related information for identifying positions of a plurality of feature points included in the first light source; The information processing device performs the camera position estimation by regarding each of the plurality of feature points as a part of the plurality of light sources based on the first position-related information. It was configured as Receiving device.

2. 2. The receiving device according to claim 1, The information processing device includes: identifying image feature points, the same number as the plurality of feature points, from within the signal region of the first wireless optical communication signal; Corresponding the image feature point to the plurality of feature points It was configured as Receiving device.

3. 3. The receiving device according to claim 2, The information processing device includes: determining a shape of the signal region of the first wireless optical communication signal, and identifying the image feature points by a method corresponding to the determined shape; Receiving device.

4. 3. The receiving device according to claim 2, The first optical wireless communication signal further comprises: The shape specifying information is information for specifying the shape of the first light source. Receiving device.

5. a receiving device equipped with a camera; an information processing device that receives, via the camera, optical wireless communication signals transmitted from each of a plurality of light sources; Equipped with The optical wireless communication signal and position-related information for identifying a position in an absolute coordinate system of a source of the optical wireless communication signal, The information processing device performs camera position estimation; The camera position estimation is identifying a signal region of the optical wireless communication signal in an image plane acquired by the camera; obtaining a light source image position indicating a position where each of the plurality of light sources is projected onto the image plane; acquiring a light source absolute position indicating a position of each of the plurality of light sources in the absolute coordinate system based on the position related information included in the optical wireless communication signal; estimating a position and orientation of the camera in the absolute coordinate system based on the light source image position and the light source absolute position; Including, a first light source included in the plurality of light sources transmits a first optical wireless communication signal; the first wireless optical communication signal includes first position-related information for identifying positions of a plurality of feature points included in the first light source; The information processing device performs the camera position estimation by regarding each of the plurality of feature points as a part of the plurality of light sources based on the first position-related information. It was configured as Optical wireless communication system.

Images