Control program and optical communication system

Abstract

To realize high speed communication in comparison with a conventional system.SOLUTION: A control program controls a receiver 3 including a color filter 32 in which a plurality of filter elements 33 corresponding to a plurality of color components are arrayed in a predetermined color array, and an imaging element 34 having a plurality of pixels arrayed corresponding to the plurality of color components. The imaging element 34 detects a luminance change equal to or greater than a threshold value as an event for each pixel, and outputs an event signal S including a pixel position where the event has occurred and a polarity of the luminance change in a case where the event is detected. The control program acquires color information of a pixel whose luminance has changed based on the event signal S, acquires a color pattern L indicating a time-series change of the color information based on a temporal change of the color information, and acquires information by decoding the color pattern L.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a control program and an optical communication system. [Background technology]

[0002] Patent Document 1 discloses an optical communication method that generates an optical signal that cyclically transitions between multiple colors and selects the first color of each symbol period from the multiple colors based on the transmission data. Patent Document 1 also discloses a technology that uses a rolling shutter camera to capture the optical signal and acquires received data based on the color obtained in each frame at a specific pixel. This allows tracking and demodulation of changes in the optical signal that are captured in only a few pixels within the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-36102 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology described in Patent Document 1, the frame rate is fixed when acquiring optical signals, which makes it difficult to read optical signals from a light source that changes pixels at high speed, and the technology does not have excellent responsiveness to high-speed color changes.

[0005] The present invention has been made in view of the above circumstances, and has as its object to realize faster communication speeds than conventional methods. [Means for solving the problem]

[0006] In order to achieve the above object, the control program of the present invention comprises: A control program for a receiver including a color filter in which a plurality of filter elements corresponding to a plurality of color components are arranged in a predetermined color array, and an imaging element having a plurality of pixels arranged corresponding to the plurality of color components, The imaging element is Detects brightness changes above a threshold for each pixel as events, When the event is detected, an event signal including the pixel position where the event occurred and the polarity of the luminance change is output; Computer, a first acquisition unit that acquires color information of pixels whose luminance has changed based on the event signal; a second acquisition unit that acquires a color pattern that indicates a time-series change in the color information based on the time change in the color information; a decoding unit that decodes the color pattern to obtain information; Function as. [Effects of the Invention]

[0007] According to the present invention, it is possible to achieve faster communication speeds than conventional methods. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an optical communication system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating color filters according to an embodiment. [Figure 3] 10A and 10B are examples of a RAW image and a color image according to an embodiment. [Figure 4] 10A and 10B are examples of a RAW image and a color image according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating an event detection process according to the embodiment. [Figure 6] 10 is a flowchart illustrating a flow of a communication process according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. As shown in Fig. 1, an optical communication system 1 according to this embodiment performs information communication by an optical communication method that utilizes changes in emitted light color. Specifically, the optical communication system 1 includes a transmitter 2, a receiver 3, and a control device 4.

[0010] The transmitter 2 is an LED (light-emitting diode) light source capable of emitting light in three colors: red, green, and blue (RGB). The transmitter 2 of this embodiment emits light in one of the RGB colors at a predetermined luminance value (e.g., 255 in 8-bit gradation) at a predetermined light-emitting interval T (e.g., 100 ms, 10 Hz cycle). This allows the transmitter 2 to emit light in a color pattern L corresponding to the communication information to be transmitted. Note that the type of lighting fixture and the light-emitting principle of the transmitter 2 are not particularly limited as long as it can emit light in the three colors: red, green, and blue. The transmitter 2 may also include, for example, a control unit that controls the color pattern L, etc.

[0011] The receiver 3 is a so-called event camera, having an image sensor that outputs only information about pixels that have experienced a luminance change as event data. Specifically, the receiver 3 outputs data indicating the polarity of the luminance change as event data, associating it with the position (pixel position) and time (timestamp) of the captured pixel where the change was detected, as part of the event detection process described below. In the receiver 3, light from the subject that has passed through the optical system 31 enters the image sensor 34 via the color filter 32. As shown in FIG. 2 , the color filter 32 has filter elements 33 corresponding to RGB color components that are two-dimensionally arranged in the xy plane, and transmits wavelengths of light corresponding to each RGB color component on a pixel-by-pixel basis. The color filter 32 in this embodiment is, for example, a Bayer-array RGB filter, with one R pixel, one B pixel, and two G pixels arranged in a 2×2 (xy) matrix. The image sensor 34 is a solid-state image sensor that detects the subject image formed through the optical system 31 and the color filter 32 and photoelectrically converts the image. The imaging element 34 has a plurality of pixels (pixel sensors) arranged in accordance with the color arrangement of the color filter 32. Each pixel observes one of the R, G, and B lights corresponding to the color arrangement of the color filter 32. Note that the receiver 3 may also include optical elements such as various filters other than those described above.

[0012] The receiver 3 configured in this manner obtains an unprocessed RAW image corresponding to the color array of the color filter 32. For example, when strong red light is incident, as shown in FIG. 3, a RAW image 61R is obtained in which the luminance value is maximum (255 in 8-bit gradation) for pixels in the image sensor 34 whose x and y coordinate values ​​are odd. By performing de-Bayering on this RAW image 61R, a color image 62R in which red appears is obtained. When strong blue light is incident, as shown in FIG. 4, a RAW image 61B is obtained in which the luminance value is maximum for pixels in the image sensor 34 whose x and y coordinate values ​​are even. By performing de-Bayering on this RAW image 61B, a color image 62B in which blue appears is obtained. However, as will be described later, in the communication processing of this embodiment, by acquiring the coordinate values ​​of pixels corresponding to the color array of the color filter 32, color changes can be identified without the need for de-Bayering or other processes to convert the image into a color image.

[0013] As shown in FIG. 1, the control device 4 is a computer that processes video (image information) acquired by the receiver 3. Specifically, the control device 4 includes an operation unit that accepts user operations and a display unit that displays various information (both not shown), as well as a storage unit 46 and a control unit 47. The storage unit 46 is a memory configured with RAM (Random Access Memory), ROM (Read Only Memory), etc., and stores various programs and data, while also functioning as a work area for the control unit 47. Specifically, the storage unit 46 stores, for example, programs for executing the communication processing described below, decoding data, etc. in advance. The control unit 47 is configured with, for example, a CPU (Central Processing Unit), etc., and controls the operation of each unit of the control device 4. For example, the control unit 47 loads a program that has been stored in advance in the storage unit 46 and executes various processes in cooperation with the loaded program.

[0014] Next, the event detection process executed by the receiver 3 of this embodiment will be described. For simplicity, only four pixels corresponding to one color array unit U (enclosed within the two-dot chain line in FIG. 2) will be described. The receiver 3 is a so-called event camera, and detects an event when the change in luminance of each pixel of the image sensor 34 exceeds a predetermined threshold. The threshold is not particularly limited, but is, for example, about 200 in 8-bit gradation. The receiver 3 asynchronously outputs the x- and y-coordinates, time, and (bright / dark) polarity of the pixel where the event was detected to the control device 4 as an event signal S(x, y, t, ±).

[0015] For example, as shown in FIG. 5, in the initial state, when the transmitter 2 emits red (R), no event signal S is output because there is no change in brightness of any pixel. Next, when the transmitter 2 emits green (G), event signals S(0, 0, 100, -), (0, 1, 100, +), and (1, 0, 100, +) are output from three pixels where a brightness change exceeding the threshold occurs. The time unit of t is [ms]. However, strictly speaking, there is a small time difference between the outputs of these event signals S. Next, when the transmitter 2 emits red (R), event signals S(0, 0, 200, +), (0, 1, 200, -), and (1, 0, 200, -) are output in a similar manner. Next, when the transmitter 2 continues to emit red (R), no event signal S is output because there is no change in brightness of any pixel. Next, when the transmitter 2 emits blue (B) light, the event signals S(0, 0, 400, -), (1, 1, 400, +) are output. In this way, the receiver 3 outputs only the data of pixels whose luminance values ​​(pixel values) have changed, thereby reducing the amount of data and achieving a high frame rate.

[0016] Next, the communication process executed by the control device 4 will be described. In the communication process, it is determined whether the event signal S from the receiver 3 is a signal of a predetermined communication method (hereinafter referred to as the "P communication method") used by the optical communication system 1, and if it is a P communication method signal, decoding process, etc. are executed. The P communication method uses a signal in which predetermined communication information is associated with a color pattern L (including a case where the same color appears consecutively) indicating a time-series change in multiple (e.g., 24) color information. The communication information is not particularly limited, but may be, for example, ID information identifying the transmitter 2. This communication process is executed by the control unit 47 reading and executing a corresponding program from the storage unit 46 based on, for example, a user's operation. Here, it is assumed that the transmitter 2 switches the RGB light emission at a light emission interval T based on a predetermined light emission command, and the receiver 3, upon receiving this light, executes the above-mentioned event detection process. It is also assumed that the transmitter 2 is fixed.

[0017] When an event detection process is executed in the receiver 3 and an event signal S is input to the control device 4, the control unit 47 first determines whether an event has occurred in multiple pixels in an area within the angle of view of the receiver 3, which is a camera (step S1). That is, the control unit 47 determines whether an event signal S has been output from multiple pixels. If the control unit 47 determines that an event has not occurred in multiple pixels (step S1; No), the control unit 47 proceeds to step S7, which will be described later.

[0018] If it is determined in step S1 that an event has occurred in multiple pixels (step S1; Yes), the control unit 47 determines whether any pixel includes an event in which the polarity of the change is "+" and the polarity of the change in the neighboring pixel is "-" (step S2). That is, the control unit 4 determines, based on the event signal S, whether the pixels corresponding to multiple color components in the same color array unit U include multiple (at least two) pixels with different polarities of brightness change. If it is determined that such an event is not included (step S2; No), the control unit 47 proceeds to step S7, which will be described later. Here, the "neighboring pixel" refers to any pixel in the same color array (in this embodiment, the Bayer array) other than the pixel with the polarity "+". That is, the determination in this step is whether the color observed by the image sensor 34 has changed.

[0019] In step S2, if it is determined that an event in which the polarity of any pixel is "+" and the polarity of its neighboring pixel is "-" is included (step S2; Yes), the control unit 47 determines whether the time interval between color changes is constant (step S3). If it is determined that the time interval between color changes is not constant (step S3; No), the control unit 47 proceeds to step S7, which will be described later.

[0020] In step S3, if it is determined that the time intervals between color changes are constant (step S3; Yes), the control unit 47 determines that the received event signal S is a signal of a predetermined communication method (P communication method) (step S4). That is, in steps S1 to S4, if an event occurs in multiple pixels, and the multiple pixels include a pixel with polarity "+" and a pixel with polarity "-" around it, and the time intervals between color changes are constant, the control unit 47 determines that the received event signal S is a signal of the P communication method.

[0021] Next, the control unit 47 buffers a predetermined number of color changes in the time direction and decodes the obtained color pattern L (step S5). Specifically, the control unit 47 buffers the color information input from the receiver 3 for, for example, 24 pieces of time (100 ms x 24 = 2.4 s). This results in a color pattern L (including cases where the same color appears consecutively) indicating time-series changes in the 24 pieces of color information. The control unit 47 then decodes the obtained color pattern L based on the decoding data to obtain the communication information. The decoding data is data that associates the color pattern L with the communication information, and is stored in advance in the storage unit 46.

[0022] Next, the control unit 47 stores the communication information acquired from the color pattern L in step S5 and the xy coordinates of the pixel within the angle of view where the light emission was detected as an event in the memory unit 46 and outputs them to a predetermined output destination (step S6).

[0023] Next, the control unit 47 determines whether or not to terminate the communication process (step S7), and if it is determined not to terminate the communication process (step S7; No), the control unit 47 proceeds to the above-mentioned step S1. On the other hand, if it is determined to terminate the communication process, for example, due to a user's termination operation or the like (step S7; Yes), the control unit 47 terminates the communication process.

[0024] As described above, according to this embodiment, the receiver 3 includes a color filter 32 having a plurality of filter elements 33 corresponding to a plurality of color components arranged in a predetermined color array, and an image sensor 34 having a plurality of pixels arranged corresponding to the plurality of color components. The control device 4 acquires color information of pixels whose luminance has changed based on an event signal S output from the image sensor 34, acquires a color pattern L based on the time change in the color information, and decodes this color pattern L to acquire information. In other words, by acquiring pixel values ​​of the image sensor 34 corresponding to the color array of the color filter, color change information can be acquired without the need for DeBayer processing or other methods to convert to a color image. This allows for suitable conversion of a group of images captured at a high frame rate. Therefore, faster communication speeds can be achieved compared to conventional methods that use a fixed frame rate. Furthermore, it is possible to switch emitted colors at speeds so fast that they are difficult for the human eye to perceive.

[0025] Furthermore, according to this embodiment, it is possible to simultaneously receive optical signals from transmitters 2 having different color change periods. That is, in conventional frame-based cameras, the camera's image acquisition period is fixed, and it is therefore only possible to receive optical signals from transmitters 2 having a color change period corresponding to that period. In contrast, according to this embodiment, the receiver 3 can acquire signals asynchronously, so that optical signals from multiple transmitters 2 that transmit optical signals with multiple different periods can be simultaneously received by a single receiver 3.

[0026] Furthermore, according to this embodiment, the image sensor 34 detects a change in luminance equal to or greater than a threshold value for each pixel as an event, and when an event is detected, outputs an event signal S as color information. By setting a large threshold value, it becomes difficult to pick up small noises, etc., and it is possible to preferably acquire only the desired optical signal.

[0027] Furthermore, according to this embodiment, in step S2 of the communication process, it is determined whether or not a plurality of pixels corresponding to a plurality of color components in the same color array include at least two pixels with different polarities of luminance change. Then, if it is determined that the plurality of pixels include at least two pixels with different polarities of luminance change, a color pattern L is acquired. That is, in step S2, it is confirmed that the acquired signal is a signal of the P communication method of the communication target, and then the color pattern L is acquired. This makes it possible to automatically identify and process the signal of the communication target. Consequently, it is possible to suppress the execution of buffering, decoding, and the like for signals other than the communication target, thereby enabling efficient communication processing.

[0028] Furthermore, according to this embodiment, if it is determined in step S2 of the communication process that the plurality of pixels includes at least two pixels with different polarities of luminance change, it is determined in step S3 whether the time interval between changes in color information is constant. If it is determined that the time interval between changes in color information is constant, color pattern L is acquired. That is, in step S3, color pattern L is acquired after more reliably confirming that the acquired signal is a signal of the P communication method of the communication target. This improves the accuracy of identifying the signal of the communication target, enabling even more efficient communication processing.

[0029] Furthermore, according to this embodiment, the communication information acquired from the color pattern L and the pixel position (coordinates) where the event occurred are stored in the storage unit 46. This makes it possible to hold desired information without the need to store image information, thereby enabling efficient use of the capacity of the storage unit 46.

[0030] It goes without saying that the embodiments to which the present invention can be applied are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit and scope of the present invention. For example, in the above-described embodiment, the transmitter 2 is fixed and does not move. However, in the communication process, the control unit 47 may determine that the transmitter 2 (light source) has moved when some of the pixels that detected (output) events (events with different polarities of luminance change) at predetermined time intervals stop outputting the events. In other words, when the transmitter 2 moves, the time change in polarity around the pixel that detected the event as a signal to be communicated becomes inconsistent, and this can be detected. In this case, the control device 4 may output a notification that the transmitter 2 has moved.

[0031] The color filter 32 may also have a layered structure in which multiple filter elements 33 corresponding to multiple color components are arranged in different layers. For example, although not particularly limited, blue may be transmitted through the first layer, green through the second layer, and red through the third layer depending on the wavelength of light. This allows the three colors to be more clearly distinguished during color discrimination. The color arrangement of the color filter 32 is not limited to the Bayer arrangement, and a wide variety of color arrangements can be applied.

[0032] The optical communication system 1 can also be used for various purposes. For example, the system can be used as a system in which a transmitter 2 is installed on a cargo handling vehicle such as a forklift, and a receiver 3 and a control device 4 are installed on the facility side to acquire the position information of the cargo handling vehicle as communication information. There are no particular limitations on the environment as long as optical signals can be used, but the system can be particularly suitable for use in environments where radio wave communication is restricted. [Explanation of symbols]

[0033] 1...optical communication system, 2...transmitter, 3...receiver, 4...control device, 32...color filter, 33...filter element, 34...imaging element, 47...control unit (first acquisition unit, second acquisition unit, decoding unit, determination unit), S...event signal

Claims

1. A control program for a receiver including a color filter in which a plurality of filter elements corresponding to a plurality of color components are arranged in a predetermined color array, and an imaging element having a plurality of pixels arranged corresponding to the plurality of color components, The imaging element is Detects brightness changes above a threshold for each pixel as events, When the event is detected, an event signal including the pixel position where the event occurred and the polarity of the luminance change is output; Computer, a first acquisition unit that acquires color information of pixels whose luminance has changed based on the event signal; a second acquisition unit that acquires a color pattern that indicates a time-series change in the color information based on the time change in the color information; a decoding unit that decodes the color pattern to obtain information; A control program that functions as a

2. the first acquisition unit determines, based on the event signal, whether a plurality of pixels corresponding to a plurality of color components in the same color array include a plurality of pixels with different polarities of luminance change; the second acquisition unit acquires the color pattern when the first acquisition unit determines that the plurality of pixels includes at least two pixels whose luminance changes have different polarities. The control program according to claim 1 .

3. when it is determined that the plurality of pixels includes at least two pixels having different polarities of luminance change, the first acquisition unit determines whether a time interval of the change in color information is constant; the second acquisition unit acquires the color pattern when it is determined that the time intervals of the changes in the color information are constant. The control program according to claim 2 .

4. the decoding unit stores the information acquired from the color pattern and the pixel position where the event occurred in a storage unit. The control program according to claim 2 .

5. the predetermined color array is a Bayer array; The control program according to claim 1 .

6. the first acquisition unit includes a determination unit that determines that a light source has moved when some of the pixels that have output the event at predetermined time intervals stop outputting the event; The control program according to claim 2 .

7. a transmitter that emits light of one of a plurality of color components at a predetermined emission interval; a receiver including a color filter in which a plurality of filter elements corresponding to the plurality of color components are arranged in a predetermined color array, and an imaging element having a plurality of pixels arranged corresponding to the plurality of color components; a control device; An optical communication system comprising: The imaging element is Detects brightness changes above a threshold for each pixel as events, When the event is detected, an event signal including the pixel position where the event occurred and the polarity of the luminance change is output; The control device acquiring color information of pixels whose luminance has changed based on the event signal; obtaining a color pattern indicating a time-series change in the color information based on the time change in the color information; decoding the color pattern to obtain information; Optical communication system.

8. The color filter has a plurality of filter elements corresponding to the plurality of color components arranged in different layers.

8. The optical communication system according to claim 7.

Images