Image processing device, optical communication device, optical communication system, program, and image processing method

Abstract

To provide an optical communication system, an image processing device, an optical communication device, a program, and an image processing method that realize optical communication that is excellent in followability and free from unpleasant blinking of a visible light source.SOLUTION: In an optical communication system 1, an image processing device 5 includes a processing unit that obtains successive images over time, obtains the positions of visible light sources in the successive images as the positions of moving objects 3a, 3b, and 3c, obtains invisible light from invisible light sources provided in the moving objects 3a, 3b, and 3c, and obtains information based on the invisible light. The image processing device 5 includes a single image sensor that acquires visible light from the visible light source and invisible light from the invisible light source.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an optical communication device, an optical communication system, a program, and an image processing method. [Background technology]

[0002] There is a known technology for transmitting information by changing a light source serving as an optical transmitting device in a predetermined lighting pattern and capturing and analyzing the lighting pattern with an imaging device serving as an optical receiving device. This type of technology is described in Patent Document 1. Patent Document 1 relates to an optical transmission device that uses a visible light LED (Light Emission Device) or an infrared light LED as a light source. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special publication 2020-523752 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a moving object is equipped with a blinking visible light source and the trajectory of the moving object is analyzed by tracking the movement of the visible light source, the blinking of the visible light source may affect the human eye and cause discomfort. While it is possible to use an invisible infrared light source instead of a visible light source, if the infrared light source blinks, the position of the infrared light source cannot be identified when the infrared light source is turned off, making it impossible to accurately track the light source using image processing. Conventional technology leaves room for improvement in terms of tracking performance.

[0005] The present invention has been made in view of the above circumstances, and has as its object to realize optical communication with excellent tracking ability and without the unpleasant flickering of a visible light source. [Means for solving the problem]

[0006] In order to achieve the above object, an image processing device of one embodiment of the present invention is characterized by comprising a processing unit that acquires successive images over time, acquires the position of a visible light source in the successive images as the position of a moving body, acquires invisible light from an invisible light source provided on the moving body, and acquires information based on the invisible light. [Effects of the Invention]

[0007] According to the optical communication device, optical communication system, program, and image processing method of the present invention, optical communication with excellent light source tracking capability can be realized without the unpleasant blinking of light. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram illustrating a configuration of an optical communication system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing a hardware configuration of the LED transmitter device of FIG. 1. FIG. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of the imaging device and image processing device of FIG. [Figure 4] 2 is a functional block diagram showing a functional configuration for a CPU of the image processing device in FIG. 1 to execute control processing. [Figure 5] 2 is a schematic diagram showing how an imaging device captures images of the movement of the moving object in FIG. 1. FIG. [Figure 6] 6 is a diagram showing an example of an image showing the movement of a bright spot of an LED of the moving object of FIG. 5. FIG. [Figure 7] FIG. 7 is a diagram showing the color of the bright spots in FIG. 6, particularly the green bright spot, in the image. [Figure 8] FIG. 7 is a diagram showing the color state of the bright spots in FIG. 6, particularly the infrared bright spots, in the image. [Figure 9] FIG. 7 is a diagram showing the color state in the image of the bright point in FIG. 6. [Figure 10] 10 is a flowchart illustrating an outline of image processing executed by the image processing apparatus of the present embodiment. [Figure 11]10 is a flowchart illustrating details of image processing, mainly decoding, executed by the image processing device of this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An optical communication system 1 according to an embodiment of the present invention will be described below with reference to the drawings.

[0010] Fig. 1 is a schematic diagram showing the configuration of an optical communication system 1 according to one embodiment of the present invention. As shown in Fig. 1, the optical communication system 1 includes an LED transmitter 2, an imaging device 4, an image processing device 5, and an operation management device 6. In this specification, the term "system" refers to an overall device made up of multiple devices, multiple means, etc.

[0011] The LED transmitter 2 is an example of an optical transmitter that uses a light source. An LED is an example of a light source, and any type of light source can be used as long as it emits light. For example, a light bulb or a laser may be used. The light source can be visible light, which is light that can be seen with the naked eye, or invisible light, which is light that cannot be seen with the naked eye. Invisible light is, for example, infrared light or ultraviolet light.

[0012] The LED transmitter 2 is an optical transmitter mounted on a moving object 3. The moving object 3 is, for example, a forklift. Three moving objects 3a to 3c are shown in FIG.

[0013] The LED transmitter 2 of this embodiment is configured to be able to control the constant lighting of the visible light for tracking and the blinking or flashing of the infrared light for transmitting information such as an identification number.

[0014] The LED transmitter 2 constantly emits visible light, and at the same time converts information to be transmitted by optical communication into changes in infrared light and emits the converted information. The position of the visible light source is acquired as the position of the mobile object 3, and information based on the invisible infrared light is acquired. The information transmitted by optical communication may be, for example, identification information for the mobile object 3. The LED transmitters 2a to 2c in FIG. 1 are arranged on the mobile objects 3a to 3c, respectively, and each transmits different identification information by optical communication.

[0015] The imaging device 4 is disposed at a position where it can capture images of the movement of the moving object 3. For example, the imaging device 4 is a digital camera. The imaging device 4 captures successive images 44 (video) over time. In the example shown in FIG. 1, the imaging device 4 has FHD (Full High Definition) pixel count (1920 pixels (pix) in the X direction and 1080 pixels (pix) in the Y direction). The identification numbers (IDs) of the three moving objects 3 (forklifts) and their positions on the images captured by the imaging device 4 are represented by pixel positions. For example, the moving object 3a at the top left in the figure is identified as having an identification number (ID) of 101, a position in the X direction of 280 out of 1920 pixels, and a position in the Y direction of 590 out of 1080 pixels.

[0016] The image processing device 5 is an information processing device that tracks the position on the image of the always-on visible light LED 29 and analyzes the blinking or flashing pattern of the infrared light LED 30 that flashes or blinks at the same position as the visible light LED 29. The imaging device 4 is used as the light receiving device because, as shown in Figure 1, it is possible to simultaneously acquire information from multiple locations from the LED transmitting device 2 and to analyze the position of the moving object 3 by converting the position on the image of the LED transmitting device 2 into actual position coordinates.

[0017] The visible light LED 29 is an example of a visible light source that emits visible light. Any type of light source can be used as long as it emits visible light. For example, a light bulb or a laser can be used. The infrared light LED 30 is an example of an infrared light source that emits infrared light. Any type of light source can be used as long as it emits infrared light. For example, a light bulb with an infrared filter or an infrared laser can be used.

[0018] The business management device 6 is a higher-level computer that analyzes the work situation and the transportation of cargo carried by the mobile object 3 based on the analysis results of the movement of the mobile object 3. In Fig. 1, the business management device 6 is configured separately from the image processing device 5, but the business management device 6 and the image processing device 5 may be configured by one computer, or may be configured by three or more computers.

[0019] Next, an example of the hardware configuration of the LED transmitter 2 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the hardware configuration of the LED transmitter 2 of Fig. 1. As shown in Fig. 2, the LED transmitter 2 includes a CPU 21, a ROM 22, a RAM 23, an acceleration sensor 24, a switch 25, a dry cell 26, a power supply controller 27, an LED driver 28, a visible light LED 29, and an infrared light LED 30.

[0020] The CPU 21 executes various processes in accordance with programs stored in the ROM 22 or programs loaded into the RAM 13. The CPU 21 is realized by a processor that executes arithmetic processing. Processors include those configured by various types of stand-alone processing devices such as a single processor, a multiprocessor, and a multicore processor, as well as those that combine these various types of processing devices with processing circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0021] The RAM 23 also stores data and the like necessary for the CPU 21 to execute various processes.

[0022] An acceleration sensor 24 detects the movement acceleration of the moving object 3. A switch 25 switches and resets the CPU 21. A dry cell 26 supplies power as a power source. A power supply controller 27 controls the power supply to an LED driver and the like.

[0023] The LED driver 28 supplies power to the visible light LED 29 and the infrared light LED 30 based on instructions from the CPU 21, and controls the lighting and blinking or flashing of the visible light LED 29 and the infrared light LED 30.

[0024] The visible light LED 29 and the infrared light LED 30 are arranged close to each other. The visible light LED 29 and the infrared light LED 30 emit light toward the surroundings by being refracted and scattered by, for example, the curved surface of the cover glass or the unevenness of the cover glass. When captured by the imaging device 4, the visible light LED 29 and the infrared light LED 30 are recognized as representing a single location.

[0025] The visible light LED 29 is always lit, and information is transmitted by utilizing the blinking or flashing of the infrared light LED 30. For example, the infrared light LED 30 emits a blinking or flashing pattern that is a lighting pattern associated with the identification number of the moving object 3 stored in the ROM 22 based on a command from the CPU 21. Note that providing redundancy to the blinking or flashing pattern of the infrared light LED 30 makes it easier to distinguish it from noise, enabling more reliable extraction of information (such as the identification number).

[0026] An example of the hardware configuration of the image processing device 5 will be described with reference to Fig. 3. As shown in Fig. 3, the image processing device 5 includes a CPU 51, a ROM 52, a RAM 53, an input unit 54, an output unit 55, a storage unit 56, and a communication unit 57.

[0027] The CPU 51 executes various processes according to a program recorded in the ROM 52 or a program loaded from the storage unit 56 into the RAM 53 .

[0028] The RAM 53 also stores data and the like necessary for the CPU 51 to execute various processes.

[0029] The CPU 51, ROM 52, and RAM 53 are interconnected via a bus. An input / output interface is also connected to this bus. The imaging device 4, an input unit 54, an output unit 55, a storage unit 56, and a communication unit 57 are connected to the input / output interface.

[0030] The input unit 54 is composed of various buttons, a microphone, etc., and inputs various information in response to user instructions. The output unit 55 is composed of a display, a speaker, etc., and outputs images 44 and sound. The storage unit 56 is composed of a hard disk or flash memory, etc., and stores data of various images 44. The communication unit 57 is a network interface that controls communication with other devices, such as the business management device 6, via a network including the Internet.

[0031] The imaging device 4 includes a color filter 41, an optical lens unit 42, and an image sensor 43. The image sensor 43 is a single sensor that can receive both visible light and invisible light, and outputs a captured image 44.

[0032] The color filter 41 is a filter that transmits red, green, and blue light, and is formed for each pixel of the image sensor 43. A combination of red, green, and blue forms one unit.

[0033] The optical lens unit 42 is composed of lenses that condense light to photograph a subject, such as a focus lens or a zoom lens. The focus lens is a lens that forms an image of the subject on the light receiving surface of the image sensor 43. The zoom lens is a lens that can freely change the focal length within a certain range. The optical lens unit 42 is provided with peripheral circuits that adjust setting parameters such as focus, exposure, and white balance as necessary.

[0034] The image sensor 43 is composed of a photoelectric conversion element, an AFE (Analog Front End), etc. The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. An object image is incident on the photoelectric conversion element through the optical lens unit 42. The photoelectric conversion element photoelectrically converts (captures) the object image, accumulates the image signal for a certain period of time, and sequentially supplies the accumulated image signal as an analog signal to the AFE. The AFE performs various signal processing on this analog image signal, such as A / D (Analog / Digital) conversion processing. A digital signal is generated by the various signal processing, and is output as an output signal of the imaging device 4. Such an output signal of the imaging device 4 is hereinafter referred to as a "captured image." Data of the captured image is appropriately supplied to the CPU 51, etc.

[0035] Next, the processing unit 60 realized by the CPU 51 of the image processing device 5 will be described with reference to Fig. 4. The processing unit 60 can also be called a processor. The processing unit 60 has an image acquisition unit 61, a tracking processing unit 62, and an analysis processing unit 63.

[0036] The image acquisition unit 61 acquires the images 44 captured by the imaging device 4. The imaging device 4 has, for example, an FHD (Full High Definition, 1920 x 1080 pixels) imaging element, and captures, for example, 30 images 44 per second. The image acquisition unit 61 acquires this digital information from the imaging device 4.

[0037] The tracking processing unit 62 identifies bright spots 45 that are always lit, particularly green bright spots 45G that are always lit green, in the image 44 acquired by the image acquisition unit 61. For example, the tracking processing unit 62 extracts only signals from green subpixels in the image 44 and finds pixels where the green signal always exceeds a predetermined threshold. The tracking processing unit 62 then tracks the green bright spots 45G, whose position continuously changes in the image 44, and identifies their positions in the image 44. For example, the tracking processing unit 62 identifies which pixel in the image 44 the bright spot 45 is located at. In the example of FIG. 1, for example, the position of the bright spot 45 on the forklift with ID 101 in the upper left corner is identified as the 280th pixel in the X direction and the 590th pixel in the Y direction.

[0038] The analysis processing unit 63 analyzes the blinking or flashing pattern of the infrared LED 30 that blinks or flashes at the same position as the green bright spot 45G extracted by the tracking processing unit 62, and identifies, for example, the identification number (ID) of the LED transmission device 2.

[0039] 5 to 9, an example of the movement of the moving object 3 and the optical signals received by the imaging device 4 and the image processing device 5 will be described. As shown in Fig. 5, the imaging device 4 captures an image of the moving object 3 moving from left to right. The LED transmitter 2 of the moving object 3 keeps the visible light LED 29 constantly lit and blinks or flashes the infrared light LED 30.

[0040] By using a green LED for the always-on visible light LED 29, it becomes easier to capture changes in infrared light. An image sensor 43 that uses a normal color filter 41 also uses an infrared cut filter. The image sensor 43 according to one embodiment of the present invention does not use an infrared cut filter, or uses a filter with weak infrared cut capability. In this case, all of the image sensors 43 that receive light that has passed through the red, green, and blue color filters 41 respond to infrared light. To achieve white balance, the blue and red outputs are typically amplified compared to green, which has a higher received light intensity.

[0041] Since there is no difference in the received light intensity for red, green, and blue infrared light, the amplified blue and red signals become relatively stronger, and infrared light produces an output signal that appears purple. For example, when infrared light is received simultaneously with green, the overall color becomes close to white. When green is lit and there is no infrared light, the color becomes green. When green is constantly lit and infrared light is flashing or blinking, the result is flashing or blinking white and green. Therefore, the combination of green and infrared light enables decoding of simple color transitions (white and green) and provides excellent robustness. In this state, the transmitter does not emit red or blue light, so the human eye only sees the constantly lit "green" color and there is no flickering, which reduces discomfort.

[0042] The constantly lit visible light LED 29 and the flashing or blinking infrared LED 30 are displayed in the image 44 as a locus of bright spot 45, as shown in FIG. 6. The visible light LED 29 is, for example, green and is recognized as a green bright spot 45G. It is constantly lit while moving, as shown in FIG. 7. To represent the green bright spot 45G, a diagonal line is drawn diagonally to the right on the green bright spot 45G on the image in FIG. 7. The infrared LED 30 is flashed or blinked, as shown in FIG. 8, and is recognized as purple by the imaging device 4, as described above. To represent the infrared bright spot 45P, a diagonal line is drawn diagonally to the left on the infrared bright spot 45P on the image in FIG. 8. As shown in FIG. 9, when the green visible light LED 29 and the infrared light LED 30 are lit, a white bright spot 45W is displayed. When the infrared light LED 30 is off, the green visible light LED 29 is always lit, and is recognized as a green bright spot 45G. To represent the white bright spot 45W, diagonal lines are drawn diagonally to the right and left in Fig. 9. As shown in Fig. 9, the bright spot 45 is recognized as a bright spot 45 that flashes or blinks between a green bright spot 45G and a white bright spot 45W.

[0043] Next, the process from acquiring the image 44 from the imaging device 4 to identifying the moving object 3 and determining its position on the image will be described with reference to Fig. 10 and Fig. 11. An overview will be provided in Fig. 10, and details related to the decoding process of the image 44 will be described in Fig. 11.

[0044] As shown in FIG. 10, when the optical communication system 1 starts, the image acquisition unit 61 of the image processing device 5 acquires the image 44 captured by the imaging device 4 (step S101).

[0045] Next, the tracking processing unit 62 extracts a constantly lit visible light bright spot 45, more specifically a green bright spot 45G, from the image 44 and performs image analysis processing to identify its position in the image 44 (step S102). In parallel, the analysis processing unit 63 performs image analysis processing to clarify information, more specifically, the identification number of the moving object 3 linked to the bright spot 45, from the blinking or blinking pattern of the infrared bright spot 45P that is blinking or flashing in the same position as the green bright spot 45G (step S102). However, once the identification number of the bright spot 45 has been clarified, there is no need to newly obtain the identification number from the blinking or blinking pattern of the infrared bright spot 45P as long as the constantly lit green bright spot 45G is being tracked.

[0046] Next, the analysis processing unit 63 performs an analysis result output process to output the analysis result to the service management device 6 through the output unit 55 of the image processing device 5 (step S103), and ends the process (step END).

[0047] This will be explained in more detail with reference to Fig. 11. The image acquisition unit 61 of the image processing device 5 performs image acquisition processing to acquire one frame of image 44 from continuously captured video (step S201). Next, the tracking processing unit 62 performs bright spot 45 extraction processing to extract bright spots 45 containing green components (step S202), and then performs direction estimation processing to estimate the moving direction of the moving object 3 from the movement history of the bright spots 45 (step S203).

[0048] After step S203, the tracking processing unit 62 performs a candidate point extraction process (step S204) in which it weights the direction of movement, extracts bright points 45 containing green components that are close to the specified movement range, and tracks the movement of the green bright point 45G. Because the light from the visible light LED 29 appears as a relatively bright bright point 45 on the imaging element, it is possible to narrow down the bright points 45 considerably by simply taking a darker image.

[0049] After step S204, the tracking processing unit 62 performs shape filtering to narrow down the candidate points that are thought to be performing optical communication, and completes the candidate point extraction process (step S204).

[0050] Furthermore, the analysis processing unit 63 performs a decoding process using the change history of the infrared light at the bright spot position (step S205). As described above, the image sensor 43 recognizes the infrared light as purple. Therefore, the change history of the infrared light in the image 44 is a change history between green and white, which is a combination of green and purple. When it is recognized as white, the infrared light is emitted.

[0051] The analysis processing unit 63 performs a pattern detection process to detect a valid light emission pattern from the decoded result (step S206). If a valid light emission pattern is detected (step S206: Yes), the analysis processing unit 63 performs a recording process to store the ID and on-screen position of the moving object 3 from the valid light emission pattern in the storage unit 56 (step S207).

[0052] If a valid light emission pattern cannot be detected from the decoded result in the detection process (step S206: No), and after the storage process (step S207) is performed, a verification process is performed to verify whether the pattern check of all bright spots 45 has been completed (step S208). If the pattern check has not been completed (step S208: No), the process returns to the bright spot extraction process (step S202) to extract bright spots 45 containing green components. If the pattern check of all bright spots 45 has been completed, the process flow ends (step END).

[0053] As described above, the image processing device 5 includes a processing unit 60 that acquires successive images 44 over time, acquires the position of the visible light LED 29 in the successive images 44 as the position of the moving body 3, acquires infrared light from the infrared light LED 30 provided on the moving body 3, and acquires information based on the infrared light.

[0054] This allows for an image processing device 5 to be obtained that achieves excellent tracking and realizes optical communication without the annoyance of the blinking of the visible light LED 29. Optical communication using a typical camera transmits information using a blinking white LED, but if the LED is turned off more frequently, it becomes difficult to track the LED. By keeping the visible light LED 29 constantly lit, tracking of the movement of the LED transmitter 2 is ensured, while a format that allows the infrared light LED 30 to blink or flash freely can be used to ensure tracking while maintaining a high coding rate. Furthermore, the infrared light of the blinking or flashing infrared light LED 30 is invisible to humans, so it does not cause annoyance.

[0055] The image processing device 5 includes a single image sensor 43 that acquires visible light from the visible light LED 29 and infrared light from the infrared light LED 30.

[0056] As a result, images of the visible light LED 29 and the infrared light LED 30 are projected onto the single image sensor 43. The images of the visible light LED 29 and the infrared light LED 30 can be arranged in the same image 44. The visible light LED 29 and the infrared light LED 30 can be linked.

[0057] The processing unit 60 of the image processing device 5 tracks the position of the green visible light LED 29G, which is always on or blinking, as the position of the moving object 3, and acquires the lighting pattern of the infrared light in the infrared region of the infrared LED 30.

[0058] This allows the imaging device 4 to use green, which is highly sensitive to light, and thus achieves excellent visibility. Furthermore, when the light is received by the imaging device 4 together with infrared light, the green signal is suppressed by the white balance adjustment circuit, and the overall image is determined to be white. A binary flashing or blinking of white and green is achieved, making analysis easy and highly reliable.

[0059] In the image processing device 5, the visible light LED 29 and the infrared light LED 30 are provided in close proximity to each other.

[0060] Visible light from the visible light LED 29 and infrared light from the infrared light LED 30 can be received by the same pixel of the image sensor 43. The infrared light from the infrared light LED 30 is received by each of the red, green, and blue sub-pixels and generates a signal. The infrared light from the infrared light LED 30 imparts color like visible light. The signal based on the infrared light and the signal based on the visible light are combined to create a color signal for the pixel consisting of red, green, and blue.

[0061] When the processing unit 60 of the image processing device 5 detects a single color represented by green at the target position, it determines that of the visible light LED 29 and the infrared light source, only the visible light LED 29 is on, and when it detects white at the target position, it determines that both the visible light LED 29 and the infrared light source are on.

[0062] This allows the visible light LED 29 to be determined as always on, making it easier to track its position. Also, the blinking or flashing of the infrared light LED 30 is extracted, and highly accurate information can be obtained by analyzing the blinking or flashing pattern.

[0063] The LED transmitter 2 is an LED transmitter 2 for optical communication mounted on a moving body 3, and is equipped with a visible light LED 29 that is controlled to light up in order to obtain the position of the moving body 3, and an infrared light LED 30 that is placed on the moving body 3 and controlled to light up so as to transmit information corresponding to the moving body 3.

[0064] As a result, the imaging device 4 receives light from the visible light LED 29 and the infrared light LED 30, thereby identifying the identification number and location of the mobile object 3 equipped with the LED transmitter 2. This allows for optical communication with excellent tracking of moving light sources without the annoying flickering of light sources. Conventional optical communication using cameras transmits information using a flickering white light source, but tracking the light source becomes difficult if the light source is turned off frequently. By keeping the visible light LED 29 constantly lit, tracking of the movement of the mobile object 3 is ensured. By using a format that allows the infrared light source to flicker or blink freely, tracking can be ensured while maintaining a high coding rate. Furthermore, because the infrared light from a flickering or blinking infrared light source is invisible to humans, it does not cause annoyance.

[0065] The optical communication system 1 comprises an LED transmitter 2 having a visible light LED 29 that is controlled to light up in order to acquire the position of a moving body 3, and an infrared light LED 30 that is placed on the moving body 3 and controlled to light up so as to transmit information corresponding to the moving body 3, and an image processing device 5 having a processing unit 60 that acquires successive images 44 over time, acquires the position of the visible light LED 29 in the successive images 44 as the position of the moving body 3, acquires infrared light from the infrared light LED 30, and acquires information based on the infrared light.

[0066] This makes it possible to realize the optical communication system 1 that has excellent tracking performance and realizes optical communication without the unpleasant blinking of the visible light LED 29.

[0067] The program causes the image processing device 5 to execute an image acquisition function for acquiring successive images 44 over time, an acquisition processing function for acquiring the position of the visible light LED 29 in the successive images 44 as the position of the moving body 3, and an analysis processing function for acquiring infrared light from the infrared light LED 30 provided on the moving body 3 and acquiring information based on the infrared light.

[0068] This makes it possible to realize a program that realizes optical communication with excellent tracking performance and without the unpleasant blinking of the visible light LED 29.

[0069] The image processing method is an image processing method executed by the image processing device 5, and includes an image acquisition step of acquiring successive images 44 over time, an acquisition processing step of acquiring the position of the visible light LED 29 in the successive images 44 as the position of the moving body 3, and an analysis processing step of acquiring infrared light from the infrared light LED 30 provided on the moving body 3 and acquiring information based on the infrared light.

[0070] This makes it possible to realize an image processing method that has excellent tracking ability and realizes optical communication without the unpleasant blinking of the visible light LED 29.

[0071] The present invention is not limited to the above-described embodiment, and any modifications and improvements that can achieve the object of the present invention are included in the present invention.

[0072] For example, the visible light LED 29 may be a visible light LED 29 that emits a color other than green. For example, a red visible light LED 29 may be used as the visible light LED 29, including as a warning. In this case, as described above, in the image capture device 4, the signal may be adjusted by an amplifier circuit of the image capture device 4 or an arithmetic circuit of the image processing device 5 so that when both the infrared light LED 30 and the red visible light LED 29 are lit, they can be recognized as white. In this case, the flashing or blinking pattern becomes a flashing or blinking of white and red at the stage of analysis processing by the image processing device 5.

[0073] As the invisible light, ultraviolet light may be used instead of infrared light. Since ultraviolet light is also invisible to humans, its blinking does not cause discomfort, and information can be obtained from the blinking pattern. In this case, the signal may be adjusted by an amplifier circuit of the imaging device 4 or an arithmetic circuit of the image processing device 5 so that when both the green LED 29G and the ultraviolet light are lit, the light is recognized as white. In this case, the blinking or flashing pattern becomes a blinking or flashing of white and green at the stage of analysis processing by the image processing device 5.

[0074] Instead of blinking infrared light, changes in the wavelength of infrared light may be used to transmit information. Since the sensitivity of the imaging device 4 depends on the wavelength of infrared light, changes in the wavelength of infrared light are recognized as changes in the intensity of the received light signal, or as differences in the intensity of the received light signals at each of the red, green, and blue sub-pixels, i.e., differences in color.

[0075] Up to this point, we have described an embodiment in which the visible light LED 29 is always on. The visible light LED 29 may have a flickering state that allows tracking without discomfort, i.e., it is normally on and turns off as appropriate. In this case, too, the same effect as the visible light LED 29 in a always-on state, namely stable tracking without discomfort, can be obtained.

[0076] In the above embodiment, the imaging device 4 to which the present invention is applied has been described as a digital camera, but is not limited to this. Also, the imaging device 4 and the image processing device 5 have been described as separate devices, but the image processing device 5 may be built into the imaging device 4.

[0077] Furthermore, each functional block in each diagram may be configured as a single piece of hardware, a single piece of software, or a combination thereof. In other words, the functional configuration in Figure 4 is merely an example and is not particularly limited. That is, it is sufficient for the image processing device 5 to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize this functionality are not particularly limited to the example in Figure 4.

[0078] Furthermore, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed into a computer or the like from a network or a recording medium.

[0079] The computer may be a computer built into dedicated hardware, or may be a computer capable of executing various functions by installing various programs, such as a general-purpose personal computer.

[0080] The recording medium containing such a program includes not only removable media distributed separately from the device main body to provide the program to the user, but also recording media provided to the user in a state where it is pre-installed in the device main body. Removable media include, for example, magnetic disks (including floppy disks), optical disks, or magneto-optical disks. Optical disks include, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), and Blu-ray (registered trademark) Discs. Magneto-optical disks include, for example, MDs (Mini-Disks). Furthermore, recording media provided to the user in a state where it is pre-installed in the device main body include, for example, the ROM 22 in FIG. 2 on which the program is recorded, or a hard disk included in the storage unit 56 in FIG. 3.

[0081] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually.

[0082] The inventions described in the claims of the present application as originally filed are set forth below. [Appendix 1] Acquire successive images over time, acquiring the position of the visible light source in the successive images as the position of the moving object; acquiring invisible light from an invisible light source provided in the moving body, and acquiring information based on the invisible light; An image processing device comprising: a processing unit. [Appendix 2] a single image sensor for acquiring the visible light from the visible light source and the invisible light from the invisible light source; 2. The image processing device according to claim 1. [Appendix 3] The processing unit tracking the position of the green visible light source, which is constantly lit or blinking, as the position of the moving object; The lighting pattern of the invisible light in the infrared region of the invisible light source is acquired. 3. The image processing device according to claim 1 or 2. [Appendix 4] The visible light source and the invisible light source are provided in close proximity to each other. 4. An image processing device according to any one of claims 1 to 3. [Appendix 5] The processing unit When green is detected at the target position, it is determined that only the visible light source is turned on among the visible light source and the infrared light source; If white light is detected at the target position, it is determined that both the visible light source and the infrared light source are on. 5. The image processing device according to claim 4. [Appendix 6] An optical transmitter for optical communication mounted on a mobile object, comprising: a visible light source that is controlled to be turned on in order to acquire the position of the moving object; an invisible light source that is disposed on the moving body and controlled to be lit so as to transmit information corresponding to the moving body; An optical transmitting device comprising: [Appendix 7] an optical transmission device having a visible light source that is controlled to be turned on in order to acquire the position of a moving object, and an invisible light source that is disposed on the moving object and controlled to be turned on so as to transmit information corresponding to the moving object; an image processing device having a processing unit that acquires successive images over time, acquires the position of the visible light source in the successive images as the position of the moving object, acquires invisible light from the invisible light source, and acquires information based on the invisible light; An optical communication system comprising: [Appendix 8] an image acquisition function for acquiring successive images over time; an acquisition processing function for acquiring the position of a visible light source in the successive images as the position of a moving object; an analysis processing function of acquiring invisible light from an invisible light source provided in the moving body and acquiring information based on the invisible light; A program that causes an image processing device to execute the above. [Appendix 9] An image processing method executed by an image processing device, an image acquisition step of acquiring successive images over time; an acquisition processing step of acquiring the position of a visible light source in the successive images as the position of a moving object; an analysis processing step of acquiring invisible light from an invisible light source provided in the moving body and acquiring information based on the invisible light; An image processing method comprising: [Explanation of symbols]

[0083] 1 Optical communication systems 2 LED transmitter 3. Mobile 4. Imaging device 5. Image processing device 6 Business management device

Claims

1. Acquire successive images over time, acquiring the position of the visible light source in the successive images as the position of the moving object; acquiring invisible light from an invisible light source provided in the moving body, and acquiring information based on the invisible light; An image processing device comprising: a processing unit.

2. a single image sensor for acquiring visible light from the visible light source and invisible light from the invisible light source; The image processing device according to claim 1 .

3. The processing unit tracking the position of the green visible light source, which is constantly lit or blinking, as the position of the moving object; The lighting pattern of the invisible light in the infrared region of the invisible light source is acquired.

3. The image processing device according to claim 1 or 2.

4. The visible light source and the invisible light source are provided in close proximity to each other.

4. The image processing device according to claim 1.

5. The processing unit When green is detected at the target position, it is determined that only the visible light source is turned on among the visible light source and the infrared light source; If white light is detected at the target position, it is determined that both the visible light source and the infrared light source are on. The image processing device according to claim 4 .

6. an optical transmission device including: a visible light source that is disposed on a moving body and is controlled to be turned on in order to acquire the position of the moving body; and an invisible light source that is disposed on the moving body and is controlled to be turned on so as to transmit information corresponding to the moving body; an image processing device having a processing unit that acquires successive images over time, acquires the position of the visible light source in the successive images as the position of the moving object, acquires invisible light from the invisible light source, and acquires information based on the invisible light; An optical communication system comprising:

7. an image acquisition function for acquiring successive images over time; an acquisition processing function for acquiring the position of a visible light source in the successive images as the position of a moving object; an analysis processing function of acquiring invisible light from an invisible light source provided in the moving body and acquiring information based on the invisible light; A program that causes an image processing device to execute the above.

8. An image processing method executed by an image processing device, an image acquisition step of acquiring successive images over time; an acquisition processing step of acquiring the position of a visible light source in the successive images as the position of a moving object; an analysis processing step of acquiring invisible light from an invisible light source provided in the moving body and acquiring information based on the invisible light; An image processing method comprising:

Images