Optical communication system, optical communication method, and information processing device

Abstract

The present disclosure relates to an optical communication system, an optical communication method, and an information processing device that make it possible to transmit and receive data stably at high speed using an image sensor. This optical communication system comprises a transmission unit and a reception unit. The transmission unit has a light emitting unit that emits a synchronization signal indicating a switching timing of data. The reception unit includes: an imaging unit that captures an image of the synchronization signal; and a timing calculation unit that acquires three or more pieces of phase information from the synchronization signal the image of which has been captured and calculates the switching timing from the acquired three or more pieces of phase information. The technology according to the present disclosure can be applied to, for example, an optical communication system that performs communication using an optical signal.

Description

Optical communication system, optical communication method, and information processing device

[0001] The present disclosure relates to an optical communication system, an optical communication method, and an information processing device, and more particularly to an optical communication system, an optical communication method, and an information processing device that use an image sensor to enable high-speed, stable data transmission and reception.

[0002] A remote controller for a television set transmits signals such as channel numbers to the receiving television set by blinking infrared light. This signal transmission method has limited application because it cannot obtain information on the light's emitting position.

[0003] In the case of optical communication using an image sensor as a light receiving unit, although it is possible to identify the light emitting position, it is difficult to improve the communication speed. For example, Patent Document 1 proposes a configuration in which a light receiving unit is provided separately from an imaging unit and synchronized in order to increase the communication speed, but this requires separate imaging units and light receiving units, and is difficult to handle when there are multiple light sources.

[0004] Non-Patent Document 1 discloses a configuration in which a transmitter having multiple LEDs transmits a synchronization signal and data. This configuration has the advantage of being able to receive multiple transmission data even when the light source frequencies are different, but the blinking cycle of the light source must be sufficiently slower than the frame rate of the image sensor, making it difficult to increase the speed.

[0005] The present applicant has proposed an optical communication method using an event sensor (EVS: event-based vision sensor) in, for example, Patent Document 2. Optical communication using an event sensor enables high-speed communication, but event sensors are currently expensive.

[0006] JP 2007-274052 A International Publication No. 2022 / 254789

[0007] “Balloon Tag: (In)visible Marker Which Tells Who's Who”, H. Aoki; S. Matsushita, Digest of Papers. Fourth International Symposium on Wearable Computers, October 16-17, 2000, Internet <URL: https: / / ieeexplore.ieee.org / document / 888491>

[0008] The present disclosure has been made in view of the above circumstances, and aims to enable high-speed, stable data transmission and reception using an image sensor.

[0009] An optical communication system according to a first aspect of the present disclosure includes a transmitting unit and a receiving unit, wherein the transmitting unit has a light emitting unit that emits a synchronization signal indicating a data switching timing, and the receiving unit has an imaging unit that captures the synchronization signal, and a timing calculation unit that acquires three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.

[0010] An optical communication method according to a second aspect of the present disclosure includes an optical communication system including a transmitter and a receiver, wherein the transmitter emits a synchronization signal indicating a data switching timing, and the receiver captures an image of the synchronization signal, obtains three or more pieces of phase information from the captured synchronization signal, and calculates the switching timing from the obtained three or more pieces of phase information.

[0011] In the first and second aspects of the present disclosure, a synchronization signal indicating the timing of switching data is emitted in a transmitting unit, the synchronization signal is imaged in the receiving unit, three or more pieces of phase information are obtained from the imaged synchronization signal, and the switching timing is calculated from the three or more pieces of phase information obtained.

[0012] An information processing device according to a third aspect of the present disclosure includes an imaging unit that captures a synchronization signal that indicates a data switching timing and is emitted as an optical signal from a predetermined light-emitting unit, and a timing calculation unit that acquires three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.

[0013] In a third aspect of the present disclosure, a synchronization signal indicating the timing of switching data is captured as an optical signal emitted from a specified light-emitting unit, three or more pieces of phase information are obtained from the captured synchronization signal, and the switching timing is calculated from the three or more pieces of phase information obtained.

[0014] The information processing device according to the third aspect of the present disclosure can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0015] The transmitting unit, receiving unit, and information processing device may each be an independent device, or may be an internal block constituting a single device.

[0016] 1 is a diagram illustrating a general optical communication technique. FIG. 1 is a diagram illustrating a general optical communication technique. FIG. 2 is a diagram illustrating a general optical communication technique. FIG. 3 is a block diagram illustrating a configuration example of an optical communication system according to a first embodiment of the present disclosure. FIG. 4 is a diagram illustrating a configuration of a light emitting unit. FIG. 5 is a diagram illustrating optical signals of a light emitting unit and an imaging unit before phase synchronization processing. FIG. 6 is a diagram illustrating a method for detecting a phase difference from light of a synchronization signal. FIG. 7 is a diagram illustrating processing for adjusting imaging timing by a timing control unit. FIG. 8 is a flowchart illustrating first phase synchronization processing performed by a receiving unit. FIG. 9 is a diagram illustrating another method for calculating a phase difference. FIG. 10 is a diagram illustrating another method for calculating a phase difference. FIG. 11 is a diagram illustrating second phase synchronization processing. FIG. 12 is a diagram illustrating second phase synchronization processing. FIG. 13 is a diagram illustrating second phase synchronization processing. FIG. 14 is a flowchart illustrating second phase synchronization processing performed by a receiving unit. FIG. 15 is a diagram illustrating correction processing of color information. FIG. 16 is a diagram illustrating an example in which the ratio of the periods of data and synchronization signal is other than 1:4. FIG. 17 is a diagram illustrating first phase detection for detecting a phase difference. FIG. 18 is a diagram illustrating second phase detection for detecting a phase difference. FIG. 10 is a diagram illustrating a second phase detection for detecting a phase difference. FIG. 11 is a diagram illustrating a third phase detection for detecting a phase difference. FIG. 12 is a diagram illustrating another configuration of a light emitting unit. FIG. 13 is a diagram illustrating a third phase synchronization process. FIG. 14 is a block diagram illustrating a configuration example of an optical communication system according to a second embodiment of the present disclosure. FIG. 15 is a diagram illustrating a first application example of the optical communication system of the present disclosure. FIG. 16 is a block diagram illustrating a configuration example of an embodiment of a computer to which the technology of the present disclosure is applied.

[0017] Hereinafter, with reference to the accompanying drawings, a description will be given of modes for carrying out the technology of the present disclosure (hereinafter referred to as embodiments). Note that in this specification and the drawings, components having substantially the same functional configurations are assigned the same reference numerals, and redundant description will be omitted. The description will be given in the following order: 1. Overview of optical communication using an image sensor 2. Example configuration of an optical communication system according to a first embodiment of the present disclosure 3. First phase synchronization processing 4. Another method for calculating a phase difference 5. Second phase synchronization processing 6. Color information correction processing 7. Phase synchronization processing other than data:synchronization signal = 1:4 8. Other example configurations of the light-emitting unit 42 9. Example configuration of an optical communication system according to a second embodiment of the present disclosure 10. Example application of an optical communication system 11. Example computer configuration

[0018] 1. Overview of Optical Communication Using Image Sensors First, an overview of optical communication using image sensors will be described with reference to FIGS. 1 to 3. FIG.

[0019] FIG. 1 is a diagram illustrating a general optical communication technique.

[0020] Optical communication is a communication method that transmits information using light such as visible light or infrared light, and is sometimes called visible light communication (VLC) or optical camera communication (OCC).

[0021] FIG. 1 shows two types of optical communication methods.

[0022] One communication method is a method of transmitting information by blinking light, called an On-Off Keying method (hereinafter referred to as the OOK method). The transmitter 1 transmits an optical signal that represents, for example, one bit of information, "1" or "0," by turning the light on (on) or off (off). The image sensor of the receiver 2 captures (receives) the optical signal from the transmitter 1 and converts (encodes) it into information according to the blinking of the light.

[0023] Another communication method is a method of transmitting information using the color of light, called Color Shift Keying (hereinafter referred to as CSK). The transmitter 1 transmits an optical signal in which, for example, two bits of "00" are represented by red, "01" by green, "10" by blue, and "11" by white. The image sensor of the receiver 2 captures (receives) the optical signal from the transmitter 1 and converts (encodes) it into two-bit information according to the color of the light.

[0024] Light is highly directional and the light-emitting position can be identified, so optical communication has the advantage of being able to send and receive information for each position.

[0025] On the other hand, in optical communications, when an image sensor is used in the receiver 2, the sampling rate is slow, for example, 60 Hz. Even if the light source frequency at which the light blinks is known, the phase of the light emission, i.e., the timing at which the light blinks or the color changes, is often unknown.

[0026] Optical communication in the case where the light source frequency is known and the phase of the light emission is unknown will be described with reference to Figures 2 and 3. Note that Figures 2 and 3 will be described using an example in which optical communication is performed using the CSK method.

[0027] The upper part of FIG. 2 shows an example in which the transmitter 1 and receiver 2 operate at the same frequency.

[0028] 2, the transmitter 1 emits light in the order of red (hereinafter referred to as R), green (hereinafter referred to as G), blue (hereinafter referred to as B), and yellow (hereinafter referred to as Y) at a light source frequency of, for example, 60 Hz. For example, R represents "00", G represents "01", B represents "10", and Y represents "11".

[0029] As in phase (1), when the receiving unit 2 can receive light at the same timing as when the emitted light color is switched in the transmitting unit 1, that is, when the timing of transmission and reception is the same, the receiving unit 2 can acquire the transmitted information without any omissions.

[0030] On the other hand, if the timing of transmission and reception is different, as in phase (2), and the receiver 2 receives light across a change in color of the optical signal, the two codes will be mixed, and the receiver 2 will not be able to correctly acquire the transmitted information.

[0031] The image sensor of the receiving unit 2 generally allows the exposure time to be set, so by shortening the exposure time, the probability of code mixing can be reduced.

[0032] The lower part of FIG. 2 shows an example in which the transmitter 1 and receiver 2 operate at the same frequency, and the image sensor of receiver 2 has a shorter exposure time.

[0033] At the reception timings of phase (4) and phase (5) in the lower part of Figure 2, the receiver 2 does not receive light across a change in color of the optical signal, so the codes are not mixed and the transmitted information can be correctly acquired.

[0034] On the other hand, at the reception timing of phase (6), the receiver 2 receives light across a color change in the optical signal, resulting in code mixing. Thus, if the transmitter 1 and receiver 2 operate at the same frequency, acquisition errors cannot be avoided unless there is a mechanism to align the phases.

[0035] On the other hand, there is also a method of correctly acquiring transmitted information by increasing the frequency of the receiver 2 higher than that of the transmitter 1 without matching the phase.

[0036] 3 shows an example in which the frequency of the receiver 2 is double that of the transmitter 1. For example, if the light source frequency of the transmitter 1 is 60 Hz, the frequency of the receiver 2 is 120 Hz.

[0037] Because the frequency of the receiver 2 is twice that of the transmitter 1, the receiver 2 receives two pieces of data for each piece of transmitted information. When the timing of transmission and reception is the same, as in phase (1), the codes do not get mixed up, so the receiver 2 can correctly obtain both pieces of received data.

[0038] On the other hand, when the phase is shifted (phase (2)), the two received data are different, so a method is required to determine which data is correct. For example, one method is to incorporate a mechanism to detect errors in codes such as parity.

[0039] Another method for determining which data is correct is to operate the receiver 2 at a frequency three times or more that of the transmitter 1 and determine the correct data by majority vote.

[0040] When the timing of transmission and reception is equal, as in phase (3) in the lower part of Figure 3, the three pieces of data received for one piece of transmitted information will be the same, and the receiver 2 can correctly acquire any piece of data.

[0041] On the other hand, at the reception timing of the phase shifted phase (4), two of the three data received for one piece of transmitted information will be correct, so the receiver 2 can determine and acquire the correct data by majority vote. However, this majority vote method increases the frequency of the receiver 2, but is not efficient.

[0042] As described above, even if the light source frequency is known, if the phase of the light emission is unknown, this hinders the speed and stability of communication. Therefore, by knowing the phase of the light emission, i.e., the timing at which the light from the transmitter 1 blinks or the color switches (hereinafter referred to as the switching timing), and synchronizing the frame of the image sensor of the receiver 2 with the light switching timing of the transmitter 1, efficient, stable communication becomes possible, enabling high-speed communication.

[0043] The following describes an optical communication system that enables efficient and stable communication by synchronizing and imaging the light emission phase when the light source frequency is known and the light emission phase is unknown. In this embodiment, synchronization means that the light source frequency and phase are the same on the transmitting side and the receiving side, and the process of synchronizing the light emission phase is called phase synchronization processing.

[0044] 2. Configuration Example of Optical Communication System According to First Embodiment of Present Disclosure FIG. 4 is a block diagram showing a configuration example of an optical communication system according to the first embodiment of the present disclosure.

[0045] The optical communication system 20 in Fig. 4 includes a transmitter 31 that transmits transmission information by optical signals, and a receiver 32 that receives the transmission information from the transmitter 31. The transmitter 31 and the receiver 32 are configured, for example, by being incorporated into separate devices. Specifically, the transmitter 31 is configured as a part of a first information processing device, and the receiver 32 is configured as a part of a second information processing device separate from the first information processing device. Note that the transmitter 31 and the receiver 32 may be incorporated into a single device, with the transmitter 31 transmitting the transmission information and the receiver 32 receiving the transmission information within the single device.

[0046] The transmitter 31 includes an encoder 41 and a light emitter 42 .

[0047] The encoder 41 encodes the transmission information, generates control information for controlling the blinking timing and light color of the light according to the encoded data, and outputs the generated control information to the light emitter 42. More specifically, the encoder 41 converts the transmission information into data represented by "0" or "1" and outputs control information according to the converted data to the light emitter 42. The control information differs depending on whether the communication method is the OOK method or the CSK method. The transmitter 31 and receiver 32 already know, based on setting information or the like, whether the communication method is the OOK method or the CSK method. The encoder 41 may add parity or the like for error correction to make the data robust against data errors and the like.

[0048] The light emitting unit 42 has a light source and causes the light source to emit light based on control information from the encoding unit 41. When the communication method is the OOK method, the light emitting unit 42 blinks, for example, white light based on the control information. On the other hand, when the communication method is the CSK method, the light emitting unit 42 blinks in a predetermined emission color, such as R (red), G (green), B (blue), or W (white), based on the control information. Note that, for simplicity, the present embodiment will be described using an example in which the light emitted by the light source is visible light, but it may also be invisible light such as infrared light.

[0049] The light-emitting unit 42 may be one or more LEDs or the like provided exclusively for transmission, or may be a light source provided in a room. Alternatively, the light-emitting unit 42 may be an organic EL display, a liquid crystal display, a projector, or the like having a two-dimensional light-emitting area.

[0050] The receiving unit 32 includes an imaging unit 51 , an image processing unit 52 , a light source detection unit 53 , a timing calculation unit 54 , a timing control unit 55 , and a decoding unit 56 .

[0051] The imaging unit 51 is composed of, for example, an image sensor, and captures (receives) the blinking timing and emitted color of the light from the light-emitting unit 42 at a predetermined frame rate. The imaging unit 51 can change the frame rate of the image sensor and the exposure time of each pixel as needed. The imaging unit 51 outputs a signal (image signal) obtained by capturing the image to the image processing unit 52. Note that in this embodiment, the light from the light-emitting unit 42 is assumed to be visible light, so the imaging unit 51 is assumed to be an image sensor that receives visible light. However, the imaging unit 51 may also be an image sensor that receives non-visible light, such as infrared light, in response to the light emitted by the light-emitting unit 42. Alternatively, the imaging unit 51 may be an event-based vision sensor (EVS) that detects changes in light intensity. Furthermore, the imaging unit 51 may have a polarizing filter attached to the front of the image sensor, or may be a polarization sensor with a polarizer. Receiving polarized light enables imaging with reflections removed.

[0052] The image processing unit 52 performs predetermined signal processing, such as correction processing and interpolation processing, on the imaging signal supplied from the imaging unit 51, converting the data into data that can be easily handled by subsequent blocks. For example, the image processing unit 52 performs signal processing such as shading correction processing and demosaic processing. The shading correction processing is processing to correct for peripheral light reduction and color shift caused by the imaging lens. In addition, for example, the image processing unit 52 performs white balance processing on the imaging signal from the imaging unit 51. When the communication method is the CSK method, color reproduction is important. Therefore, the white balance processing performed by the image processing unit 52 differs from general white balance processing that corrects for an ambient light source, and can perform optimal white balance processing and color matrix processing to acquire color information from the light-emitting unit 42.

[0053] The light source detection unit 53 detects an area (transmission information area) that is emitting transmission information from the image acquired by the imaging unit 51, and outputs the detected area to the timing calculation unit 54. For example, the light source detection unit 53 identifies the transmission information area based on information such as the brightness, color, arrangement, shape, and blinking cycle of the light emitted by the light emitting unit 42. A pattern unique to the transmission information area may be detected by pattern matching or the like. If the device having the transmission unit 31 is a mobile device, the light source detection unit 53 also has a function of detecting and tracking the movement of the light emitting unit 42. The light source detection unit 53 outputs the identified blinking and color changes of the light of the transmission information to the timing calculation unit 54 and the decoding unit 56.

[0054] As a method for easily detecting the transmission information area by the light source detection unit 53, a specific blinking or color change pattern may be determined to make the detection easier. Alternatively, the transmission information area on the display of the transmitter 31 may be fixed to make the detection easier.

[0055] The timing calculation unit 54 calculates the switching timing, which is the timing at which the light in the transmission information area blinks or changes color, supplied from the light source detection unit 53, and outputs timing information indicating the switching timing to the timing control unit 55. That is, the timing calculation unit 54 calculates the timing for matching and synchronizing the phase of the light emitted in the transmission information area. If the light in the transmission information area is divided into synchronization signal light and data light, the timing calculation unit 54 calculates the switching timing based on the synchronization signal light. The timing calculation unit 54 also controls the exposure time based on the emission brightness of the transmission information area so as not to saturate the pixels of the imaging unit 51. The timing information output by the timing calculation unit 54 to the timing control unit 55 also includes information used by the timing control unit 55 to determine the exposure time.

[0056] The timing control unit 55 outputs a control signal for adjusting the image capture timing to the image capture unit 51 based on the timing information calculated by the timing calculation unit 54. The image capture timing includes exposure timing (shutter timing) and exposure time. The timing control unit 55 controls the image capture unit 51 so as to synchronize the light emission phase with that of the light emitting unit 42.

[0057] The decoding unit 56 decodes the optical signal into transmission information based on the blinking of the light or changes in color. If the transmission information includes correction information such as parity for error correction, the decoding unit 56 has a function of correcting the error and restoring the data to normal. The decoding unit 56 also has a function of outputting an error or the like if decoding is not possible due to a change in the distance or angle between the transmitting unit 31 and the receiving unit 32. The decoding unit 56 outputs the transmission information obtained by decoding to a subsequent block (not shown) as reception information. For example, the reception information is stored in a storage unit or output to a processing unit that processes the reception signal.

[0058] The optical communication system 20 has the above configuration, and transmit information is converted into an optical signal, output from the transmitter 31, and received by the receiver 32. The receiver 32 captures the optical signal and decodes it into transmit information.

[0059] The light source frequency when the transmitter 31 outputs an optical signal is stored as setting information, for example, inside each of the transmitter 31 and the receiver 32, and is shared. Alternatively, the receiver 32 may be configured to be able to obtain information about the light source frequency from the arrangement of LEDs, an image pattern displayed on a display, or the like.

[0060] The information transmitted and received between the transmitting unit 31 and the receiving unit 32 may be, for example, light source ID information that identifies the light source, text information such as a URL (Uniform Resource Locator), environmental information such as temperature and air pressure, audio information, image information, etc.

[0061] When the information processing device equipped with the transmitter 31 or the receiver 32 is, for example, a mobile object (e.g., a robot, a drone, etc.) that moves in water, in the air, on land, etc., a mouse, a game controller, etc., the transmitter 31 and the receiver 32 transmit and receive, for example, light source ID information that individually identifies each light source as transmission information. This allows the device on the receiver 32 side to, for example, estimate its own position based on the light source ID information.

[0062] When the information processing device equipped with the transmitter 31 or the receiver 32 is an IoT device used in, for example, livestock farming, agriculture, marine aquaculture, etc., the transmitter 31 and the receiver 32 transmit and receive, as transmission information, location information of the installation location where the device on the transmitter 31 side is installed, and measurement information (environmental information) measured at the installation location, such as temperature, air pressure, humidity, gas, and flow rate. This enables the device on the receiver 32 side to perform, for example, remote sensing.

[0063] When the information processing device equipped with the transmitter 31 or the receiver 32 is, for example, a mobile terminal such as a smartphone or tablet, the transmitter 31 and the receiver 32 transmit and receive information such as sales information, detailed information, and authentication information, etc., of the device on the transmitter 31 side, such as advertisements, URLs, text information, and two-dimensional codes. Optical communication can be used for advertisements, authentication, etc. Furthermore, by utilizing the directionality of light, it is possible to transmit audio and images only to terminals in specific locations.

[0064] 3. First Phase Synchronization Processing First, the first phase synchronization processing performed by the transmitting unit 31 and the receiving unit 32 will be described with reference to FIGS.

[0065] In the first phase synchronization process, the timing at which the light blinks and the color changes in the multiple light sources of the light-emitting unit 42 is the same, and the image sensor that constitutes the imaging unit 51 is described as capturing images using a global shutter method in which all pixels capture images at the same timing.

[0066] FIG. 5 is a diagram illustrating the configuration of the light emitting unit 42 and the optical signal transmitted by the light emitting unit 42. As shown in FIG.

[0067] 5, the light-emitting unit 42 has eight light sources 71. Each light source 71 is formed, for example, by an LED. Of the eight light sources 71, one light source 71, specifically light source 71-0, is a synchronization signal transmitter that transmits a synchronization signal indicating the timing of switching data, and the remaining seven light sources 71, specifically light sources 71-1 to 71-7, are data transmitters that transmit data.

[0068] The light source 71-0, which serves as a synchronization signal transmitter, transmits a synchronization signal by repeatedly blinking at a known light source frequency. The seven light sources 71, which serve as data transmitters, transmit data using the CSK method, which represents two bits of information using four color information bits.

[0069] In this embodiment, the light sources 71-1 to 71-7 emit light to represent two bits of "00" as R (red), "01" as G (green), "10" as B (blue), and "11" as Wh (white). The light-emitting unit 42 has seven light sources 71 as a data transmission unit, and can transmit 14 bits of data at a time. If data is transmitted using the OOK method, each of the seven light sources 71 represents "1" by turning on and "0" by turning off, and 7 bits of data are transmitted at a time.

[0070] The eight light sources 71 of the light-emitting unit 42 are arranged asymmetrically in the vertical and horizontal directions, so that the synchronization signal transmission unit and the data transmission unit can be identified even if the imaging unit 51 is rotated at a predetermined angle when capturing an image. However, if the orientation of the light-emitting unit 42 is known, the light sources 71 may be arranged symmetrically. For example, if the entire light-emitting unit 42 is composed of an LED display, its installation location is fixed, and its orientation is known to be a predetermined orientation, the light sources 71 may be arranged symmetrically. If the device on the receiving unit 32 side is a mobile object or the like and the orientation of the imaging unit 51 may change, the device on the receiving unit 32 side may be provided with an inertial sensor (IMU) or the like to detect tilt, etc., to control the orientation of the imaging unit 51 or to change the position of each light source 71 captured in the image.

[0071] The lower part of FIG. 5 shows examples of optical signals transmitted by the light source 71-0 of the synchronization signal transmitting section and the light sources 71-i (i=1 to 7) of the data transmitting section.

[0072] In this embodiment, the light source 71-0 of the synchronization signal transmission unit transmits a synchronization signal at a light source frequency of 15 Hz. The light source 71-i of the data transmission unit transmits data at a light source frequency of 60 Hz. In other words, the synchronization signal emits light at a frequency that is 1 / 4 of the data rate. These light source frequencies are shared with the receiving unit 32 by some means.

[0073] The data rate of 60 Hz is a typical imaging frame rate for the imaging unit 51, which is composed of an image sensor. The imaging unit 51 acquires four frames of data, thereby acquiring one cycle of data for the synchronization signal. The blinking light of the synchronization signal is usually a triangular wave, but it can be approximated as a sine wave. In Figure 5, the black portion of the first half cycle of one cycle of the synchronization signal represents the negative part of the sine wave, and the white portion represents the positive part of the sine wave.

[0074] FIG. 6 shows the optical signals of the light emitting unit 42 and the imaging unit 51 before phase synchronization processing.

[0075] The light emitting unit 42 transmits R ("00"), G ("01"), B ("10"), Wh ("11"), and B ("10") at a data rate of 60 Hz.

[0076] The imaging unit 51 captures the synchronization signal and data before phase synchronization processing, with the phase shifted by φ. Of the four frames of data acquired by the imaging unit 51, the first frame is captured in a color (yellow) that is a mixture of R and G. The second frame is captured in a color (cyan) that is a mixture of G and B. The third and fourth frames are captured in a color (light blue) that is a mixture of B and Wh.

[0077] On the other hand, when the imaging unit 51 acquires four frames of data, it can acquire phase information of four points at 90-degree intervals within one cycle of the synchronization signal. Of the four frames of data, the luminance of the synchronization signal acquired at the timing of the first frame is designated L0, the luminance of the synchronization signal acquired at the timing of the second frame is designated L1, the luminance of the synchronization signal acquired at the timing of the third frame is designated L2, and the luminance of the synchronization signal acquired at the timing of the fourth frame is designated L3.

[0078] FIG. 7 shows the blinking of light of a synchronization signal that approximates a sine wave, and the luminances L0, L1, L2, and L3, which are four phase information of the synchronization signal obtained at the timing of the first to fourth frames.

[0079] In FIG. 7, θ0, θ1, θ2, and θ3 are sampling phase angles, which represent 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively.

[0080] If the blinking of the light of the synchronization signal is approximated as a sine wave, the phase difference φ can be obtained using the luminances L0, L1, L2, and L3, which are the four pieces of phase information acquired within one period.

[0081] Specifically, the synchronization signal approximated to a sine wave in FIG. 7 can be expressed by the following equation (1): L(θ)=A·sin(θ)+B·cos(θ)+C (1)

[0082] A, B, and C in equation (1) are parameters that represent a sine wave. When expressed as a matrix, the parameter x can be calculated by the following equation (2) using the least squares method.

[0083] Using the luminance values ​​L0, L1, L2, and L3 obtained in four frames, the parameters A, B, and C can be calculated as follows:

[0084] Equation (1) can be expressed as equation (3) by converting trigonometric functions. In equation (3), φ represents a phase difference. The phase difference φ is expressed by the following equation (4) using the luminance values ​​L0, L1, L2, and L3 obtained in four frames.

[0085] As described above, when four pieces of phase information, namely luminance L0, L1, L2, and L3, of the synchronization signal are acquired over four frames, the phase difference φ can be calculated using equation (4).Furthermore, if at least three pieces of phase information can be acquired, the phase difference φ can be calculated using equation (2).

[0086] The timing calculation unit 54 calculates the phase difference φ based on the luminance values ​​L0, L1, L2, and L3, which are four pieces of phase information of the synchronization signal supplied from the light source detection unit 53, and outputs the calculated phase difference φ as timing information to the timing control unit 55. Since the unit of the phase difference φ in equation (4) is radian, the phase difference φ is converted into time information based on real-time information such as the frame rate and is supplied to the timing control unit 55.

[0087] FIG. 8 is a diagram illustrating a process in which the timing control unit 55 adjusts the image capturing timing based on the timing information from the timing calculation unit 54.

[0088] The timing control unit 55 adjusts the imaging timing based on the timing information from the timing calculation unit 54 so as to synchronize the phase with that of the light emitting unit 42. That is, the timing control unit 55 controls the imaging unit 51 so as to delay the start of imaging by a time δ corresponding to the phase difference φ. The time δ is a waiting time corresponding to the phase difference φ, and after adjusting the time δ, the imaging unit 51 can perform imaging in phase synchronization with the light emission of the light emitting unit 42.

[0089] 9, the first phase synchronization process performed by the receiver 32 will be described. This process is started, for example, when an optical signal serving as transmission information is output from the light emitter 42 of the transmitter 31.

[0090] First, in step S11, the imaging unit 51 captures four frames of data and a synchronization signal. The four frames of data and the synchronization signal are subjected to appropriate predetermined signal processing by the image processing unit 52 and the light source detection unit 53, and then supplied from the light source detection unit 53 to the timing calculation unit 54.

[0091] In step S12, the timing calculation unit 54 calculates the phase difference φ based on the luminances L0, L1, L2, and L3, which are four pieces of phase information of the synchronization signal supplied from the light source detection unit 53. The phase difference φ is converted into time information and then output to the timing control unit 55 as timing information.

[0092] In step S13, the timing control unit 55 adjusts the imaging timing based on the timing information from the timing calculation unit 54 to match the phase with the light emitting unit 42. That is, the timing control unit 55 controls the imaging unit 51 to delay the start of imaging by a time δ corresponding to the phase difference φ. Since the image sensor constituting the imaging unit 51 captures images using the global shutter method, adjusting the imaging timing adjusts the switching timing commonly for all pixels.

[0093] In step S14, the imaging unit 51 waits for a time δ and then starts imaging.

[0094] After the first phase synchronization process is completed, the image capturing unit 51 can capture images in phase with the light emitted by the light emitting unit 42, and can transmit and receive data stably at high speed.

[0095] <4. Another Method for Calculating Phase Difference> Although it has been explained that the phase difference φ can be found by the above-mentioned formula (2) or formula (4), the phase difference φ can also be calculated by another calculation method described below.

[0096] Another method for calculating the phase difference φ will be described with reference to FIGS.

[0097] FIG. 10 is a diagram for explaining a method of calculating the phase difference φ when light is emitted at a frequency of a synchronization signal that is ¼ of the data rate.

[0098] 10 , if the time of one frame, which is the data cycle, is T, the transmitter 31 transmits a synchronization signal having one cycle of four frames (4T). For example, as one cycle of the synchronization signal, a signal of "0" (off), "0" (off), "1" (on), "1" (on) is transmitted every T time. The imaging unit 51 of the receiver 32 acquires four phase information of the synchronization signal over four frames. Of the four phase information of the synchronization signal acquired over four frames, namely luminances L0, L1, L2, and L3, the data with the lowest signal level (luminance) is assumed to be luminance L0.

[0099] Among the luminances L0, L1, L2, and L3, when luminance L0 has the lowest signal level, luminance L0 is data acquired when the synchronization signal is "0" (black portion) throughout the entire exposure time, and luminance L2 is data acquired when the synchronization signal is "1" (white portion) throughout the entire exposure time. Luminances L1 and L3 correspond to the phase difference (phase shift) φ. More specifically, as shown in A of FIG. 10, luminance L1 is the minimum luminance state when the light is off (i.e., luminance L0) plus the maximum luminance state when the light is on (i.e., luminance L2) by a ratio of φ / T. Similarly, luminance L3 is the maximum luminance state when the light is on (i.e., luminance L2) plus the minimum luminance state when the light is off (i.e., luminance L0) by a ratio of φ / T. Therefore, luminances L1 and L3 can be formulated using the phase difference φ as shown in Equations (5) and (6). L1(φ) = L0 + (L2 - L0) × φ / T ・・・・・・(5) L3(φ) = L2 - (L2 - L0) × φ / T ・・・・・・(6)

[0100] Equations (5) and (6) can be graphed as shown in Figure 10B. As can be seen from the graph in Figure 10B, there is a relationship of L0 + L2 = L1 + L3.

[0101] Transforming equations (5) and (6) into an equation for calculating the phase difference φ gives: φ = T × (L1 - L0) / (L2 - L0) .....(5)' = T × (L2 - L3) / (L2 - L0) .....(6)' Therefore, the phase difference φ can be calculated using either equation (5)' or equation (6)'. The phase difference φ may be calculated using both equations (5)' and (6') and the average value may be used as the final phase difference φ.

[0102] FIG. 11 is a diagram for explaining a method of calculating the phase difference φ when light is emitted at a frequency of a synchronization signal that is 1 / 3 of the data rate.

[0103] 11 , the transmitter 31 transmits a synchronization signal having one cycle of three frames (3T). For example, as one cycle of the synchronization signal, a signal of "0" (off), "0" (off), and "1" (on) is transmitted every T time periods. The imaging unit 51 of the receiver 32 acquires three pieces of phase information of the synchronization signal over three frames. Of the three pieces of phase information of the synchronization signal acquired over three frames, namely luminance L0, L1, and L2, the data with the lowest signal level (luminance) is assumed to be luminance L0.

[0104] In this case, the luminances L1 and L2 are also data corresponding to the phase difference φ. As shown in A of FIG. 11, the luminance L1 is the state of the minimum luminance when the light is off (i.e., luminance L0) plus the state of the maximum luminance when the light is on, by a ratio of φ / T. The luminance L2 is the state of the maximum luminance when the light is on plus the state of the minimum luminance when the light is off (i.e., luminance L0) by a ratio of φ / T. The maximum luminance when the light is on can be expressed as (L1 + L2 - L0), and the luminances L1 and L2 can be graphed as shown in B of FIG. 11. The phase difference φ can be calculated using the following equation (7): φ = T × (L1 - L0) / (L1 + L2 - L0) (7)

[0105] As described above, even in the other methods for calculating the phase difference φ, the phase difference φ can be calculated by obtaining three or more pieces of phase information of the synchronization signal, as in the above-mentioned equations (5)′, (6)′, and (7).

[0106] 5. Second Phase Synchronization Processing Next, the second phase synchronization processing performed by the transmitting unit 31 and the receiving unit 32 will be described with reference to FIGS.

[0107] In the second phase synchronization process, it is assumed that the transmitter 31 is configured as part of a smartphone or personal computer, and the light emitter 42 is configured as a display that updates the displayed image using sequential scanning. When the image is updated using sequential scanning, the timing of display switching differs between the top and bottom of the image displayed on the display. The imaging unit 51 that captures the image displayed on the display is similar to the above example in that it is configured as an image sensor, but captures the image using a rolling shutter method that starts exposure line-sequentially from the top row, rather than a global shutter method.

[0108] FIG. 12 shows an example of transmission information displayed on the display serving as the light-emitting unit 42. In FIG.

[0109] The display serving as the light-emitting unit 42 displays transmission information in a predetermined area 101. Of the images displayed on the display, the area 101 in which the transmission information is displayed is referred to as the transmission information area 101. The transmission information area 101 includes a data area 111 for displaying data and a synchronization signal area 112 for displaying a synchronization signal.

[0110] 12, the data area 111 includes 4x4=16 light sources 121, and each light source 121 emits light, for example, representing two bits of "00" as R (red), "01" as G (green), "10" as B (blue), and "11" as Wh (white). In this case, the entire data area 111 can transmit 32 bits of data. One light source 121 is made up of multiple pixels of the display.

[0111] The synchronization signal area 112 has four light sources 122 arranged to correspond to each of the 4x4 rows of the data area 111. In other words, because the image displayed on the display is updated by sequential scanning, the display switching timing differs for each row of the data area 111. Therefore, the synchronization signal area 112 displays the switching timing individually for each row of the data area 111. One light source 122 is composed of multiple pixels on the display. Note that the light source patterns of the data area 111 and the synchronization signal area 112 do not necessarily need to match. For example, as shown in the examples of Figures 13 to 16, a pattern arrangement can be used that makes it easier to detect the transmission information area 101, such as making the light sources 122 of the synchronization signal area 112 linear compared to the square light sources 121 of the data area 111.

[0112] FIG. 13 is a diagram for explaining the imaging of the transmission information area 101 displayed on the display by the imaging unit 51. In FIG.

[0113] 13, the display, which is the light-emitting unit 42, displays transmission information in a predetermined area of ​​the display (transmission information area 101). At this time, the time during which the display displays the transmission information in the transmission information area 101 by sequential scanning is assumed to be time T1.

[0114] The image sensor serving as the imaging unit 51 performs imaging operations at 60 Hz, the same frame rate as the display, and captures the transmission information area 101 displayed on the display. Even if the frame rate and vertical synchronization timing are the same, the position and size of the transmission information area 101 appearing in the captured image 141 will vary depending on the distance between the imaging unit 51 (image sensor) and the light-emitting unit 42 (display), the angle of view of the imaging unit 51, and other factors. For example, if the imaging unit 51 is far from the light-emitting unit 42, the size of the transmission information area 101 will be small, as shown in the middle of Figure 13. Conversely, if the imaging unit 51 is close to the light-emitting unit 42, the size of the transmission information area 101 will be large, as shown in the bottom of Figure 13.

[0115] 13, when the transmission information area 101 of the light-emitting unit 42 appears small in the captured image 141 captured by the imaging unit 51, the time taken by the imaging unit 51 to capture the transmission information area 101 using the rolling shutter method is assumed to be time T2, which is shorter than time T1. On the other hand, when the transmission information area 101 of the light-emitting unit 42 appears large in the captured image 141, as shown in the bottom row of FIG. 13, the time taken by the imaging unit 51 to capture the transmission information area 101 using the rolling shutter method is assumed to be time T3, which is longer than time T1.

[0116] The light source detection unit 53 of the receiving unit 32 detects the transmission information area 101 from the image captured by the imaging unit 51. For example, the light source detection unit 53 identifies the transmission information area 101 based on information such as a predetermined pattern shape, brightness, and color that serves as a landmark within the transmission information area 101, and outputs the identified area to the timing calculation unit 54. The timing calculation unit 54 generates timing information for adjusting the imaging timing based on the position and size of the transmission information area 101, and outputs the generated timing information to the timing control unit 55.

[0117] FIG. 14 corresponds to the example shown in the middle of FIG. 13 and is a diagram for explaining a control example when the transmission information area 101 appears small in the captured image 141.

[0118] 14, the timing calculation unit 54 generates timing information for adjusting the imaging timing so that the imaging time of the transmission information area 101 in the captured image 141 becomes the same as the display time T1 of the transmission information area 101 in the light-emitting unit 42. Specifically, the timing calculation unit 54 limits the rows to be exposed so that only an area 142 in the vertical direction in the captured image 141 that corresponds to the transmission information area 101 is imaged, and generates timing information for lengthening the horizontal synchronization timing that controls the exposure time of one row.

[0119] FIG. 15 is a timing chart illustrating the control for lengthening the horizontal synchronization timing.

[0120] 15 shows normal imaging, in which the imaging unit 51 is driven to capture an image of the entire pixel array, and imaging is performed based on a vertical timing signal VT and a horizontal timing signal HT. The vertical timing signal VT is a synchronization signal that controls switching between frames, and one frame period (vertical scanning period) is time V1. The horizontal timing signal HT is a synchronization signal that controls the readout of one row of the pixel array, and one row period (horizontal scanning period) is time H1.

[0121] In contrast, the vertical timing signal VT' and horizontal timing signal HT' in the lower part of Figure 15 show the driving of the imaging unit 51 after the timing calculation unit 54 has adjusted the timing so that only the area 142 of the transmission information area 101 in the captured image 141 is captured.

[0122] For simplicity's sake, let's assume that the four rows of light sources 121 and 122 in the transmission information area 101 in the captured image 141 correspond in size to four rows of the pixel array of the imaging unit 51. The imaging unit 51 only needs to read out four rows during the time V1 of one frame. Therefore, the imaging unit 51 performs partial driving in the vertical direction of the pixel array, capturing only four rows corresponding to the region 142 of the transmission information area 101. In other words, the imaging unit 51 does not drive the pixel rows above and below the region 142 of the transmission information area 101. Furthermore, in the horizontal direction of the pixel array, the imaging unit 51 reads out signals by changing the readout time for one row from time H1 to time H2. Time H2 is longer than time H1 when all pixels of the imaging unit 51 are read out. The vertical timing signal VT' is the same as the vertical timing signal VT.

[0123] FIG. 16 corresponds to the example shown in the lower part of FIG. 13 and is a diagram illustrating a control example in which the transmission information area 101 is shown large in the captured image 141.

[0124] When the transmission information area 101 in the captured image 141 is large and the time T3 for capturing the transmission information area 101 is longer than the display time T1 of the transmission information area 101 on the light-emitting unit 42, the timing calculation unit 54 generates timing information to adjust the capture timing so that the capture time of the transmission information area 101 in the captured image 141 is equal to the display time T1 of the transmission information area 101 on the light-emitting unit 42, as shown in the lower part of FIG. 16 . Specifically, since the readout time of one frame needs to be shortened, the timing calculation unit 54 generates timing information to perform thinning readout, in which predetermined rows of the pixel array are thinned out and read. This shortens the readout time of the transmission information area 101 by the imaging unit 51 from time T3 to time T1. When the transmission information area 101 is large in the captured image 141, even if predetermined rows are thinned out, the light sources 121 and 122 of each row of the transmission information area 101 are included in the rows that are not thinned out, so the transmission information can be acquired.

[0125] Next, the second phase synchronization process performed by the receiving unit 32 will be described with reference to the flowchart of Fig. 17. This process is started, for example, after an image showing the transmission information is displayed on the display, which is the light-emitting unit 42.

[0126] First, in step S31, the imaging unit 51 captures an image of a subject including a display, which is the light-emitting unit 42. The image signal is appropriately subjected to predetermined signal processing in the image processing unit 52, and then supplied to the light source detection unit 53.

[0127] In step S32, the light source detection unit 53 identifies the transmission information area 101 in the captured image based on the mark in the transmission information area 101.

[0128] In step S33, the timing calculation unit 54 generates timing information for adjusting the imaging timing based on the size of the transmission information area 101 identified by the light source detection unit 53, and outputs the generated timing information to the timing control unit 55. If the transmission information area 101 of the light emitting unit 42 appears small, timing information is generated to limit the driving area in the vertical direction and lengthen the horizontal synchronization timing. Conversely, if the transmission information area 101 appears large, timing information is generated to perform thinning readout by thinning out some rows in the vertical direction.

[0129] In step S34, the timing control section 55 outputs a control signal for adjusting the image capturing timing to the image capturing section 51 based on the timing information from the timing calculation section 54.

[0130] In step S35, the imaging unit 51 starts imaging based on the control signal and generates an image that matches the transmission information area 101.

[0131] In step S36, the timing calculation unit 54 generates timing information for phase synchronization based on the light sources 122 in the synchronization signal area 112 of the transmission information area 101 of the captured image after the imaging timing adjustment, and outputs the generated timing information to the timing control unit 55. The timing control unit 55 outputs a control signal for adjusting the switching timing to the imaging unit 51 based on the timing information from the timing calculation unit 54. That is, after the imaging timing is adjusted in accordance with the size of the transmission information area 101, a phase difference φ is detected based on the synchronization signal in the transmission information area 101 of the captured image, and a phase alignment process is performed. In other words, in step S36, the above-described first phase synchronization process is performed for each row of the light sources 121 and 122 in the transmission information area 101.

[0132] After the second phase synchronization process is completed, the image capturing unit 51 can capture an image synchronized with the image of the transmission information area 101 showing the transmission information on the display, which is the light emitting unit 42, and can transmit and receive data stably at high speed.

[0133] 6. Color Information Correction Processing When the light-emitting unit 42 is configured as a display, the range of colors that can be reproduced (color gamut) generally varies depending on the product or individual display. The transmission information area 101 may include color adjustment information that corrects variations in the color gamut of such individual displays.

[0134] FIG. 18 is a diagram showing an example of the transmission information area 101 that displays color adjustment information.

[0135] 18 includes a color information area 113 in addition to a data area 111 and a synchronization signal area 112. The color information area 113 includes a red reference light source 123R, a green reference light source 123G, and a blue reference light source 123B.

[0136] The image processing unit 52 acquires the RGB values ​​of the red reference light source 123R, the green reference light source 123G, and the blue reference light source 123B in the color information area 113, and calculates the correction matrix M of equation (8) for color correction.

[0137] In equation (8), [R, G, B] are the RGB values ​​of the red reference light source 123R, the green reference light source 123G, and the blue reference light source 123B obtained from the captured image, and [R', G', B'] are the RGB values ​​of the red reference light source 123R, the green reference light source 123G, and the blue reference light source 123B after color correction.

[0138] The correction matrix M can be calculated as follows.

[0139] Now, the RGB value of the red reference light source 123R obtained from the captured image is (L RR , L RG , L RB ), the RGB value of the green reference light source 123G is (L GR , L GG , L GB ), the RGB values ​​of the blue reference light source 123B are (L BR , L BG , L BB ) When the red reference light source 123R, the green reference light source 123G, and the blue reference light source 123B are each captured by a plurality of pixels, (L RR , L RG , L RB ), (L GR, L GG , L GB ), (L BR , L BG , L BB ) is the average value of the multiple pixels.

[0140] The image processing unit 52 calculates the red reference light source 123R using the following equation (9): RR , L RG , L RB ) was normalized (L RR ', L RG ', L RB ').

[0141] Similarly, for the green reference light source 123G and the blue reference light source 123B, the image processing unit 52 calculates (L GR , L GG , L GB ) was normalized (L GR ', L GG ', L GB '), (L BR , L BG , L BB ) was normalized (L BR ', L BG ', L BB ').

[0142] Then, the image processing unit 52 calculates the normalized RGB values ​​(L RR ', L RG ', L RB '), RGB values ​​of green reference light source 123G (L GR ', L GG ', L GB '), RGB values ​​of blue reference light source 123B (L BR ', L BG ', L BB '), the correction matrix M is calculated by the inverse matrix calculation of equation (10).

[0143] After the correction matrix M is calculated, the color information of each light source 121 in the data area 111 is corrected based on the correction matrix M and decoded.

[0144] 7. Phase Synchronization Processing Other Than Data:Synchronization Signal = 1:4 In the above example, the phase synchronization processing was described for the case where the data frequency is 60 Hz and the synchronization signal frequency is 15 Hz, in other words, the synchronization signal emits light at a frequency that is ¼ of the data rate, as shown in A of Fig. 19. In this case, four pieces of phase information can be obtained by driving at 60 Hz for one cycle of the 15 Hz synchronization signal, so the receiving unit 32 can calculate the phase difference φ using the above-mentioned equation (4) and adjust the switching timing.

[0145] Next, we will explain phase synchronization processing when the frequency ratio of data to synchronization signal is other than 1:4. Cases where the frequency ratio of data to synchronization signal is other than 1:4 include, for example, a case where the data frequency is 60 Hz, the synchronization signal frequency is 30 Hz, and the data:synchronization signal ratio is 1:2, as shown in B of Fig. 19, or a case where the frequencies of the data and synchronization signal are the same, i.e., a case where the data frequency is 60 Hz, the synchronization signal frequency is 60 Hz, and the data:synchronization signal ratio is 1:1, as shown in C of Fig. 19. Note that even in these cases, the frequencies of the data and synchronization signal are known between the transmitter 31 and the receiver 32.

[0146] As described above, to detect the phase difference φ, it is necessary to obtain at least three pieces of phase information for one cycle of the synchronization signal, but in the cases of B and C in Figure 19, only two or one pieces of phase information can be obtained for one cycle of the synchronization signal, so a separate drive is required to detect the phase difference φ.

[0147] FIG. 20 is a diagram illustrating the first phase detection for detecting the phase difference φ.

[0148] The receiving unit 32 has a phase detection mode for acquiring phase information of the synchronization signal. In the phase detection mode, the imaging unit 51 captures images at a frequency that differs by a predetermined ratio from the frequency of a known synchronization signal, as shown in FIG. 20 . The timing control unit 55 acquires three or more pieces of phase information from three or more images captured by the imaging unit 51. In the example of FIG. 20 , the imaging unit 51 captures images at 52.5 Hz, which is 7 / 8 of the 60 Hz frequency of the synchronization signal emitted by the light emitting unit 42. By acquiring at least three pieces of phase information in the phase detection mode, the phase difference φ can be determined and timing information can be generated.

[0149] As another example of the first phase detection, in the phase detection mode, the imaging unit 51 may increase the frame rate by reducing the number of pixels read out by performing pixel addition or pixel thinning drive, and may acquire at least three pieces of phase information for one cycle of the synchronization signal.

[0150] After the phase difference φ is detected in the above-described phase detection mode, the timing control unit 55 controls the imaging unit 51 to delay the start of imaging by a time δ corresponding to the phase difference φ, thereby enabling imaging that is phase-synchronized with the light emission of the light-emitting unit 42.

[0151] FIG. 21 is a diagram illustrating the second phase detection for detecting the phase difference φ.

[0152] As shown in FIG. 21 , the image sensor constituting the imaging unit 51 can drive a pixel group consisting of four pixels in a 2x2 (2 rows and 2 columns) pixel array to have different exposure timings (shutter timings) and exposure times for the four pixels in the pixel group. This function is called a quad shutter control function (https: / / www.sony-semicon.com / ja / products / is / industry / gs / imx900.html). The color filters are configured so that the four pixels in a pixel group receive light of the same wavelength, for example, in a quad Bayer array in which the color filters are arranged in a Bayer array for each pixel group. A monochrome image sensor without color filters may also be used.

[0153] When the image sensor constituting the imaging unit 51 has such a quad shutter control function, by varying the exposure timing for the four pixels, pixel A to pixel D, in the pixel group, as shown in Fig. 22, it is possible to obtain four pieces of phase information for pixels A to D with respect to one cycle of the synchronization signal. In other words, the pixel signals obtained by pixel A, pixel B, pixel C, and pixel D correspond to the above-mentioned luminances L0, L1, L2, and L3, respectively, and the phase difference φ can be calculated using equation (4). The exposure times for pixels A to D are the same.

[0154] Although not shown in the figure, for example, if the exposure timing of the four pixels in a pixel group is divided into the upper two pixels of pixel A and pixel B and the lower two pixels of pixel C and pixel D, then it is possible to obtain brightness L0 from the upper two pixels of the first frame, brightness L1 from the lower two pixels, brightness L2 from the upper two pixels of the second frame, and brightness L3 from the lower two pixels, and the phase difference φ can be calculated over two frames.

[0155] When the color filters are not in a quad Bayer array of 2x2 units but in a normal Bayer array, as shown in Figure 23, by varying the exposure timing for two pixels, pixel B and pixel C, in which G filters of the same color (G1, G2) are arranged, it is possible to obtain four pieces of phase difference information in two frames and calculate the phase difference φ.

[0156] FIG. 24 is a diagram illustrating the third phase detection for detecting the phase difference φ.

[0157] The third phase detection method involves driving a quad-bayer image sensor with quad shutter control functionality to obtain four phase information values ​​for pixels A to D in response to one synchronization signal cycle by varying the exposure timing and exposure time of each of the four pixels in a pixel group. For example, the imaging unit 51 controls the exposure times of pixels A to D so that the ratio of their exposure times is 1:2:3:4. Assuming that the luminance values ​​obtained by pixels A to D, each equipped with an R color filter, are R0, R1, R2, and R3, as shown in FIG. 24, the timing calculation unit 54 can obtain phase information by calculating the difference between the pixel signals. Specifically, pixel signals with the same exposure time can be obtained by setting luminance values ​​R1' = R1 - R0, R2' = R2 - R1, and R3' = R3 - R2. Luminance R0' is the same as luminance R0. Therefore, for one cycle of the synchronization signal, four pieces of phase information R0', R1', R2', and R3' can be obtained from pixels A to D that have R color filters, and the phase difference φ can be calculated. Similar calculations can be performed using pixels A to D that have G and B color filters.

[0158] As described above, even when the frequencies of the data and the synchronization signal are the same or when the cycles of the data and the synchronization signal are one to two, by using the quad shutter control function of the image sensor, it is possible to obtain two or more pieces of phase information from one frame of image, more specifically, from two or more pixels in one frame of image that have different exposure timings, thereby making it possible to detect the phase difference φ and adjust the switching timing.

[0159] The phase difference φ may be obtained by appropriately combining the first to third phase detections described above.

[0160] 8. Other Configuration Examples of the Light Emitting Section 42 FIG. 25 is a diagram illustrating another configuration of the light emitting section 42 and an optical signal transmitted by the light emitting section 42. In FIG.

[0161] 5, the eight light sources 71 are divided into one light source 71 that transmits a synchronization signal and seven light sources 71 that transmit data. In contrast, the light-emitting unit 42′ in FIG. 25 has eight light sources 71, and all eight light sources 71 are light sources that transmit both data and synchronization signals.

[0162] Specifically, as shown in the lower part of Fig. 25, the synchronization signal is expressed by the blinking cycle, and the data is expressed by the color information when the blinking is turned on. As in Fig. 5, the color information is expressed by 2 bits of "00" represented by R (red), "01" by G (green), "10" by B (blue), and "11" by Wh (white), and when emitting light, 16 bits of data can be transmitted at once using eight light sources 71.

[0163] Referring to FIG. 26, the phase synchronization process (third phase synchronization process) when capturing an image of an optical signal from the light emitting unit 42' in FIG. 25 will be described.

[0164] Under the control of the timing control unit 55, the imaging unit 51 captures multiple frames of the optical signal from the light-emitting unit 42 while shifting the phase by a predetermined amount, such as phase (1), phase (2), phase (3), and so on, as shown in the upper part of FIG. 26 , during a predetermined period during which phase synchronization processing is performed. When the luminance of multiple frames is observed, the luminance is observed to be maximum at the center in the time direction when the light-emitting unit 42′ is turned on, as shown in the lower part of FIG. 26 . The timing calculation unit 54 calculates the timing at which the luminance is maximum from the luminance observed in multiple frames and detects the phase of the transmitted optical signal. In this case, the phase can be aligned without calculating the above-described equations (2) and (4). It is also possible to calculate the phase difference φ from the luminance of multiple captured frames using equations (2) and (4), or the phase difference φ may be calculated using equations (2) and (4).

[0165] 9. Configuration Example of Optical Communication System According to Second Embodiment of Present Disclosure FIG. 27 is a block diagram illustrating a configuration example of an optical communication system according to a second embodiment of the present disclosure.

[0166] In the second embodiment of FIG. 27, the parts corresponding to those in the first embodiment shown in FIG. 4 are given the same reference numerals, and the description of those parts will be omitted.

[0167] In the first embodiment described above, the receiving unit 32 detects the phase difference φ based on the optical signal received from the transmitting unit 31 and synchronizes the phases. In contrast, in the second embodiment, the transmitting unit 31 and the receiving unit 32 acquire common time information from the outside and perform transmitting and receiving operations based on the acquired time information, thereby performing phase-synchronized communication.

[0168] The optical communication system 20 in Fig. 27 includes a transmitting unit 31 that transmits transmission information by optical signals, a receiving unit 32 that receives the transmission information from the transmitting unit 31, and a time information generating device 201. The transmitting unit 31 and the time information generating device 201 are connected to each other via a predetermined network such as a LAN (Local Area Network), the Internet, a telephone line network, or a satellite communication network.

[0169] The time information generating device 201 generates time information that serves as a reference for transmission and reception timing, and outputs the time information to the transmitting unit 31 and the receiving unit 32. The time information generating device 201 is composed of an NTP (Network Time Protocol) server that distributes accurate time information, or a server that implements the PTP (Precision Time Protocol) defined in the IEEE-1588 standard. The time information generating device 201 may also be a GNSS (Global Navigation Satellite System) satellite that transmits a GNSS signal containing highly accurate time information, such as a GPS signal.

[0170] The light emitting unit 42' of the transmitting unit 31 acquires time information from the time information generating device 201, and determines the transmission timing of an optical signal as transmission information based on the time information, and transmits the optical signal. Except for the fact that the transmission timing is determined based on the time information from the time information generating device 201, the light emitting unit 42' is the same as that of the first embodiment described above.

[0171] The timing control unit 55' of the receiving unit 32 acquires time information from the time information generating device 201 and determines the timing of receiving the optical signal from the transmitting unit 31. The timing control unit 55' outputs a control signal to the imaging unit 51 to synchronize the data switching timing with that of the transmitting unit 31. The receiving unit 32 of the second embodiment does not include the timing calculation unit 54.

[0172] The optical communication system 20 has the above configuration, and the transmitting unit 31 and the receiving unit 32 acquire common time information from the time information generating device 201, and perform transmitting and receiving operations based on the acquired time information, thereby enabling phase-synchronized communication.

[0173] 10. Application Examples of Optical Communication System An example in which the above-described optical communication system is applied will be described.

[0174] FIG. 28 shows a first application example of the optical communication system of the present disclosure, in which the optical communication system is applied to communication between smartphones.

[0175] Smartphone 301 is a transmitting device that transmits predetermined transmission information, and includes a display 311 as a transmitting unit 31. Smartphone 302 is a receiving device that receives the transmission information from smartphone 301, and includes a camera 321 on the rear side as an imaging unit 51 of receiving unit 32. Smartphone 301 on the transmitting side displays transmission information 312 on display 311. Smartphone 302 on the receiving side captures the transmission information 312 displayed on display 311 with camera 321.

[0176] According to communication between smartphones using the optical communication system of the present disclosure, communication is carried out within the range of visible light, so it is possible to communicate with desired parties while simplifying security and authentication processes.

[0177] FIG. 29 shows a second application example of the optical communication system of the present disclosure, in which the optical communication system is applied to vehicle-to-vehicle communication and vehicle-to-infrastructure communication.

[0178] Vehicle 351 is a transmitting device that transmits predetermined transmission information and includes a transmitting unit 31 (not shown). Traffic light 352, which is a roadside device, is also a transmitting device that transmits predetermined transmission information and includes a transmitting unit 31 (not shown). Vehicle 353 is a receiving device that receives transmission information from vehicle 351 or traffic light 352 and includes a receiving unit 32 (not shown). Vehicle 351 on the transmitting side transmits an optical signal that carries information such as its own position and speed as transmission information. Traffic light 352 transmits an optical signal that carries information such as signal information, regulation information, road information, and environmental information. Vehicle 353 on the receiving side receives the transmission information from vehicle 351 or traffic light 352 and uses it for safe driving assistance and autonomous driving assistance.

[0179] As described above, the optical communication system of the present disclosure can be used for safe driving assistance and autonomous driving assistance in vehicle-to-vehicle communication and road-to-vehicle communication.

[0180] 11. Example of Computer Configuration The series of processes executed by the transmitter 31 or the receiver 32 can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed in a computer. Here, the term "computer" includes microcomputers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0181] FIG. 30 is a block diagram showing an example of the hardware configuration of a computer as an information processing device when a series of processes executed by the transmitting unit 31 or the receiving unit 32 is executed by a program.

[0182] The computer 400 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, and a RAM (Random Access Memory) 403. The CPU 401, the ROM 402, and the RAM 403 are connected to one another by a bus 404.

[0183] An input / output interface 405 is further connected to the bus 404. An input unit 406, an output unit 407, a storage unit 408, a communication unit 409, and a drive 410 are connected to the input / output interface 405.

[0184] The input unit 406 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 407 includes a display, speaker, output terminal, etc. The storage unit 408 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 409 includes a network interface, etc. The drive 410 drives removable media 411 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0185] In the computer 400 configured as above, the CPU 401 performs the above-described series of processes by, for example, loading a program stored in the storage unit 408 into the RAM 403 via the input / output interface 405 and the bus 404 and executing the program. The RAM 403 also stores data and the like necessary for the CPU 401 to execute various processes as appropriate.

[0186] The program executed by the CPU 401 of the computer 400 can be provided by being recorded on a removable medium 411 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0187] In the computer 400, the program can be installed in the storage unit 408 via the input / output interface 405 by inserting the removable medium 411 into the drive 410. The program can also be received by the communication unit 409 via a wired or wireless transmission medium and installed in the storage unit 408. Alternatively, the program can be installed in the ROM 402 or the storage unit 408 in advance.

[0188] The program executed by computer 400 may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0189] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0190] For example, it is possible to adopt a configuration in which all or part of the above-described embodiments are combined.

[0191] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0192] Furthermore, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. When one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0193] In this specification, the steps described in the flowcharts may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0194] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0195] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0196] The technology disclosed herein may employ the following configurations: (1) An optical communication system including a transmitter and a receiver, wherein the transmitter has a light-emitting unit that emits a synchronization signal indicating data switching timing, and the receiver has an imaging unit that captures the synchronization signal, and a timing calculation unit that acquires three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information. (2) The optical communication system described in (1), wherein the imaging unit captures the synchronization signal at a predetermined frame rate, and the timing calculation unit acquires one piece of phase information from an image of one frame and calculates the switching timing from the three or more pieces of phase information acquired over multiple frames. (3) The optical communication system described in (1), wherein the imaging unit captures the synchronization signal at a predetermined frame rate, and the timing calculation unit acquires two or more pieces of phase information from an image of one frame and calculates the switching timing from the three or more pieces of phase information acquired over multiple frames. (4) The optical communication system according to (3), wherein the timing calculation unit acquires two or more pieces of phase information using two or more pixels with different exposure timings in one frame image and calculates the switching timing. (5) The optical communication system according to (4), wherein the timing calculation unit acquires two pieces of phase information using two pixels with different exposure timings in one frame image and calculates the switching timing from the four pieces of phase information acquired in the two frame images. (6) The optical communication system according to (4), wherein the timing calculation unit acquires four pieces of phase information using four pixels with different exposure timings in one frame image and calculates the switching timing from the four pieces of phase information acquired in the one frame image. (7) The optical communication system according to (6), wherein the phase information is a pixel signal acquired by the pixel. (8) The optical communication system according to (6), wherein the phase information is a difference in pixel signal between a predetermined pixel and one of the four pixels. (9) The optical communication system according to any one of (1) to (8), wherein the timing calculation unit calculates the switching timing by calculating a phase difference from the synchronization signal from the acquired three or more pieces of phase information.(10) The optical communication system according to any one of (1) to (9), wherein the imaging unit captures an image including the data and the synchronization signal using a rolling shutter system, and the timing calculation unit generates timing information for lengthening horizontal synchronization timing in accordance with a size of an area in the image including the data and the synchronization signal, and acquires the three or more pieces of phase information from the synchronization signal of the image whose imaging timing has been adjusted by the timing information. (11) The optical communication system according to any one of (1) to (10), wherein the imaging unit captures an image including the data and the synchronization signal using a rolling shutter system, and the timing calculation unit generates timing information in accordance with a size of an area in the image including the data and the synchronization signal so as to capture only an area whose vertical direction corresponds to the area, and acquires the three or more pieces of phase information from the synchronization signal of the image whose imaging timing has been adjusted by the timing information. (12) The optical communication system according to any one of (1) to (9), wherein the imaging unit captures an image including the data and the synchronization signal using a rolling shutter system, the timing calculation unit generates timing information to perform thinning readout by thinning out a predetermined number of rows according to a size of an area including the data and the synchronization signal in the image, and acquires the three or more pieces of phase information from the synchronization signal of the image whose imaging timing has been adjusted by the timing information. (13) The optical communication system according to any one of (1) to (12), wherein the imaging unit captures the synchronization signal at a frequency that differs by a predetermined ratio from the frequency of the synchronization signal, and the timing calculation unit acquires the three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information. (14) The optical communication system according to any one of (1) to (13), wherein the imaging unit captures the synchronization signal by increasing a frame rate, and the timing calculation unit acquires the three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.(15) The optical communication system according to any one of (1) to (14), wherein the imaging unit images the synchronization signal while shifting the phase by a predetermined amount, and the timing calculation unit calculates the switching timing from the brightness of the imaged synchronization signals. (16) The optical communication system according to any one of (1) to (15), wherein the light emitting unit has a light source that emits an optical signal representing the data and a light source that emits an optical signal representing the synchronization signal. (17) The optical communication system according to any one of (1) to (15), wherein the light emitting unit has a light source that emits optical signals representing the data and the synchronization signal. (18) An optical communication method of an optical communication system including a transmitting unit and a receiving unit, comprising: the transmitting unit emitting a synchronization signal that indicates data switching timing; and the receiving unit imaging the synchronization signal; and acquiring three or more pieces of phase information from the imaged synchronization signal and calculating the switching timing from the acquired three or more pieces of phase information. (19) An information processing device having an imaging unit that captures a synchronization signal that is emitted as an optical signal from a predetermined light-emitting unit and indicates a data switching timing, and a timing calculation unit that acquires three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information. (20) The information processing device according to (19), wherein the imaging unit captures an image including the data and the synchronization signal.

[0197] 20 Optical communication system, 31 Transmitter, 32 Receiver, 41 Encoder, 42, 42' Light emitter, 51 Imaging unit, 52 Image processor, 53 Light source detector, 54 Timing calculator, 55, 55' Timing controller, 56 Decoder, 71 Light source, 101 Transmission information area, 111 Data area, 112 Synchronization signal area, 113 Color information area, 121 Light source, 122 Light source, 201 Time information generating device, 301 Smartphone, 302 Smartphone, 311 Display, 312 Transmission information, 321 Camera, 351 Vehicle, 352 Traffic light, 353 Vehicle, 401 CPU, 402 ROM, 403 RAM, 406 Input unit, 407 Output unit 408 storage unit, 409 communication unit, 410 drive

Claims

1. An optical communication system comprising a transmitting unit and a receiving unit, wherein the transmitting unit has a light emitting unit that emits a synchronization signal indicating the timing of switching data, and the receiving unit has an imaging unit that captures the synchronization signal, and a timing calculation unit that acquires three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.

2. The optical communication system according to claim 1, wherein the imaging unit captures the synchronization signal at a predetermined frame rate, and the timing calculation unit obtains one piece of phase information from one frame of image and calculates the switching timing from the three or more pieces of phase information obtained over multiple frames.

3. The optical communication system according to claim 1, wherein the imaging unit captures the synchronization signal at a predetermined frame rate, and the timing calculation unit acquires two or more pieces of phase information from one frame of image and calculates the switching timing from the three or more pieces of phase information acquired over multiple frames.

4. An optical communication system according to claim 3, wherein the timing calculation unit obtains two or more pieces of phase information using two or more pixels with different exposure timings within one frame image, and calculates the switching timing.

5. The optical communication system according to claim 4, wherein the timing calculation unit obtains two pieces of phase information using two pixels with different exposure timings within one frame of image, and calculates the switching timing from the four pieces of phase information obtained from the two frame images.

6. The optical communication system according to claim 4, wherein the timing calculation unit acquires four pieces of phase information using four pixels with different exposure timings within one frame of image, and calculates the switching timing from the four pieces of phase information acquired from one frame of image.

7. The optical communication system according to claim 6, wherein the phase information is a pixel signal acquired by the pixel.

8. An optical communication system according to claim 6, wherein the phase information is a difference between a pixel signal of a predetermined pixel among the four pixels.

9. The optical communication system according to claim 1, wherein the timing calculation unit calculates the switching timing by calculating the phase difference with the synchronization signal from the three or more pieces of phase information acquired.

10. The optical communication system of claim 1, wherein the imaging unit captures an image including the data and the synchronization signal using a rolling shutter method, and the timing calculation unit generates timing information that lengthens the horizontal synchronization timing depending on the size of an area in the image that includes the data and the synchronization signal, and obtains the three or more pieces of phase information from the synchronization signal of the image whose imaging timing has been adjusted using the timing information.

11. The optical communication system of claim 1, wherein the imaging unit captures an image including the data and the synchronization signal using a rolling shutter method, and the timing calculation unit generates timing information according to the size of an area in the image including the data and the synchronization signal so that only an area in the vertical direction corresponding to the area is captured, and the three or more pieces of phase information are obtained from the synchronization signal of the image whose imaging timing has been adjusted by the timing information.

12. The optical communication system of claim 1, wherein the imaging unit captures an image including the data and the synchronization signal using a rolling shutter method, and the timing calculation unit generates timing information to thin out and read out a predetermined number of rows depending on the size of an area including the data and the synchronization signal in the image, and obtains the three or more pieces of phase information from the synchronization signal of the image whose imaging timing has been adjusted using the timing information.

13. The optical communication system of claim 1, wherein the imaging unit images the synchronization signal at a frequency that is different by a predetermined ratio from the frequency of the synchronization signal, and the timing calculation unit acquires the three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.

14. The optical communication system of claim 1, wherein the imaging unit images the synchronization signal at an increased frame rate, and the timing calculation unit acquires the three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.

15. The optical communication system according to claim 1, wherein the imaging unit images the synchronization signal while shifting the phase by a predetermined amount, and the timing calculation unit calculates the switching timing from the brightness of the multiple synchronization signals that have been imaged.

16. The optical communication system according to claim 1, wherein the light emitting unit has a light source that emits an optical signal representing the data, and a light source that emits an optical signal representing the synchronization signal.

17. The optical communication system according to claim 1, wherein the light emitting unit has a light source that emits an optical signal representing the data and the synchronization signal.

18. An optical communication method comprising: an optical communication system including a transmitter and a receiver; the transmitter emitting a synchronization signal indicating data switching timing; and the receiver capturing an image of the synchronization signal; acquiring three or more pieces of phase information from the captured synchronization signal; and calculating the switching timing from the acquired three or more pieces of phase information.

19. An information processing device having an imaging unit that captures a synchronization signal that indicates the timing of switching data and is emitted as an optical signal from a specified light-emitting unit, and a timing calculation unit that acquires three or more pieces of phase information from the captured synchronization signal and calculates the switching timing from the acquired three or more pieces of phase information.

20. The information processing device according to claim 19, wherein the imaging unit captures an image including the data and the synchronization signal.

Images