Processing system, processing method, and processing program

Abstract

Provided are a processing system, processing method, and processing program that make it possible to detect a prescribed area of an object as a marker without attaching a marker. This processing system comprises an emission unit for emitting emission light onto an object, an imaging unit for imaging radiated light that has been radiated from the object in response to the emission of the emission light, a detection unit for using an image captured by the imaging unit to detect the radiated light as a marker for the object, and a processing unit for carrying out prescribed processing on the basis of the radiated light detected by the detection unit.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Application No. 2020-040479, filed on March 10, 2020, the contents of which are incorporated herein by reference. [Technical Field]

[0002] The present invention relates to a processing system, a processing method, and a processing program. [Background technology]

[0003] Conventionally, so-called object tracking, which uses computer vision to acquire position information and orientation information of an object, has been utilized in various situations. A common method for such object tracking is, for example, a method in which feature points of an object are detected by image processing and the detected feature points are tracked to estimate the position information and orientation information of the object (see Non-Patent Document 1). However, with this method, the detection of feature points is affected by the shape, texture, or movement of the object, and therefore the detection accuracy of the feature points may be insufficient. To address this issue, there is a method of attaching a marker serving as a characteristic point to the object in advance. One example of such a marker is a marker (active marker) that can be detected by emitting light itself. For example, Patent Document 1 listed below discloses a method of positioning a load suspended from a lifting rope by arranging multiple light-emitting diodes serving as markers on the load and capturing the light emitted by the multiple light-emitting diodes with a video camera. Furthermore, Patent Document 2 listed below discloses a configuration for detecting the amount of movement of a conveyor belt in an inkjet recording device, in which a phosphorescent material is applied to the surface of the conveyor belt in advance, and a predetermined pattern of light is irradiated onto the applied area, causing the irradiated portion of the phosphorescent material to emit light and be used as a marker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 5-505788 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-66460 [Non-patent literature]

[0005] [Non-Patent Document 1] Alper Yilmaz, Omar Javed, and Mubarak Shah, “Object Tracking: A Survey,” ACM Computing Surveys, Vol.38, No.4, Article 13, (2006) Summary of the Invention

[0006] However, the configurations disclosed in Patent Documents 1 and 2 require the prior effort of installing or applying a light source or a phosphorescent substance that can serve as a marker to the object, and this results in changes to the appearance or shape of the object. On the other hand, if an attempt is made to detect feature points of an object without using a marker, the detection accuracy of the feature points may be insufficient, as described above.

[0007] Therefore, the present invention provides a processing system, processing method, and processing program that can use a specified area of ​​an object as a marker without installing or applying anything that can be used as a marker to the object. [Means for solving the problem]

[0008] A processing system according to one aspect of the present invention includes an irradiation unit that irradiates an object with irradiation light, an imaging unit that captures emitted light emitted from the object itself in response to the irradiation of the irradiation light, a detection unit that detects the emitted light as a marker of the object based on the image captured by the imaging unit, and a processing unit that executes predetermined processing based on the emitted light detected by the detection unit.

[0009] According to this aspect, by using the emitted light emitted from the object itself when irradiated with irradiation light as a marker, it is possible to detect a predetermined area of ​​the object as a marker and execute a predetermined process without changing the appearance or shape of the object or impairing the mobility of the object. Note that in this specification, irradiating the object with irradiation light and using the emitted light emitted from the object itself in the irradiated area as a marker is sometimes referred to as "adding an emitted light marker."

[0010] In the above aspect, the processing unit may measure at least one of a relative position, a relative orientation, and a relative speed of the object with respect to the imaging unit, and track the object.

[0011] According to this aspect, it is possible to measure at least one of the relative position, relative orientation, and relative velocity of an object and track the object without installing or applying anything that can be used as a marker to the object.

[0012] In the above aspect, the imaging unit may capture, at multiple timings, emitted light emitted from the object itself in response to a certain irradiation, the detection unit may detect the emitted light from each of the multiple images captured at the multiple timings, and the processing unit may calculate the relative speed of the object with respect to the imaging unit based on the change in the position of the emitted light between the multiple images.

[0013] According to this aspect, the relative velocity of the object with respect to the imaging unit is calculated based on the change in the position of the emitted light emitted in response to a certain irradiation, so that the object can be tracked regardless of the distance between the irradiation unit and the imaging unit.

[0014] In the above aspect, the imaging unit captures an image including multiple emitted lights each emitted from the object itself in response to irradiation over multiple timings, the detection unit detects the multiple emitted lights from the image, and the processing unit may calculate the relative speed of the object with respect to the imaging unit based on the intervals between the multiple emitted lights.

[0015] According to this aspect, the relative speed of the object with respect to the imaging unit is calculated based on the intervals between multiple emitted lights emitted in response to irradiation over multiple timings, so that the object can be tracked regardless of the distance between the irradiation unit and the imaging unit.

[0016] In the above aspect, the irradiation unit continuously irradiates irradiation light, the imaging unit captures an image including emitted light emitted from the object itself in response to the continuous irradiation, the detection unit detects the trajectory of the emitted light, and the processing unit may calculate the relative speed of the object with respect to the imaging unit based on the detected trajectory of the emitted light.

[0017] According to this aspect, the relative speed of the object with respect to the imaging unit is calculated based on the trajectory of the emitted light, so that the object can be tracked at a rate that is finer than the frame rate of the imaging unit.

[0018] In the above aspect, the imaging unit may acquire a multi-viewpoint image including the emitted light, and the processing unit may measure the shape of the object based on the emitted light included in the acquired multi-viewpoint image.

[0019] According to this aspect, even if the object moves in an unknown and random manner, new emitted light markers can be continuously added in real time, so that the shape of the object can be measured in a simple manner without the markers affecting the measurement. Furthermore, the shape of an object can be measured even if its shape changes.

[0020] In the above aspect, the processing unit may visualize a trajectory of movement of the object using the detected emitted light.

[0021] According to this aspect, even if the object is a liquid, gas, powder, or the like, the trajectory of the movement of these objects can be visualized.

[0022] In the above aspect, the processing unit may calculate a trajectory of movement of the object based on the detected emitted light, and correct motion blur in the image including the object based on the calculated trajectory.

[0023] According to this aspect, it is possible to reduce the calculation cost and obtain highly accurate results compared to when performing blind deconvolution without using a blur kernel. Since the velocity of the object can be calculated from the luminance of the emitted light, correction can be performed even if the object accelerates or decelerates during the exposure time of the imaging device.

[0024] In the above aspect, the processing unit may perform motion capture based on the detected emitted light.

[0025] According to this aspect, posture estimation can be performed without damaging the appearance. Dynamic movements can be accommodated with fewer constraints. Since the emitted light contains time information, posture estimation can be performed with higher accuracy than with physical markers. Furthermore, the position and timing of attaching the emitted light marker can be selected.

[0026] In the above aspect, the processing unit may perform simultaneous localization and mapping (SLAM) based on emitted light detected from an object in the surrounding environment of the processing system.

[0027] According to this aspect, by using the light emitted from the object itself in the surrounding environment as a marker, the emitted light used as the marker can be accurately detected even if the viewpoint of the photograph is different, thereby improving measurement accuracy.

[0028] In the above aspect, the processing unit may measure a mechanical property of the object based on the trajectory of the detected emitted light.

[0029] According to this aspect, there is no need to attach physical markers, strain sensors, etc. to the object, so measurements can be made without affecting the mechanical properties of the object, improving the measurement accuracy of mechanical properties, etc.

[0030] In the above aspect, the irradiation unit may irradiate irradiation light along a boundary surface separating at least two regions, and the processing unit may determine whether the object has touched the boundary surface based on emitted light emitted from the object upon contact with the irradiated light.

[0031] According to this aspect, it is possible to automatically attach an emitted light marker to a predetermined object with a relatively simple configuration of irradiating the object with irradiation light.

[0032] In the above aspect, the processing system may further include a light path control unit that guides the irradiation light emitted from the irradiation unit to a desired region of the object.

[0033] According to this aspect, even if the object moves relative to the irradiating unit, it becomes easier to continue irradiating a specific area of ​​the object with the irradiating light. Also, even if the object is small in size or there are restrictions on irradiation on the surface of the object, it becomes easier to attach the emitting light marker to the object.

[0034] In the above aspect, the imaging unit may include a high-speed camera that captures the emitted light at a frame rate of 100 fps or more.

[0035] According to this aspect, if the emission duration of the emitted light is several tens of milliseconds or more, the emitted light emitted from the object can be captured across multiple images, and therefore the emitted light can be used as a marker for many objects without the need for applying a phosphorescent material or the like.

[0036] In the above aspect, the image capturing section may capture an image of emitted light emitted from the object itself after irradiation of the irradiation light onto the object has stopped.

[0037] According to this aspect, when capturing an image of emitted light, it is possible to distinguish between irradiated light and emitted light, and therefore it is possible to detect the emitted light as a marker with high accuracy.

[0038] In the above-described embodiment, the emitted light may include delayed fluorescence, phosphorescence, afterglow, phosphorescence, or infrared light emitted from the object itself after irradiation with the irradiating light.

[0039] According to this aspect, emitted light is emitted due to excitation of electrons in the object, so that the emitted light can be detected robustly regardless of the appearance or shape of the object.

[0040] A processing method according to another aspect of the present invention includes irradiating an object with irradiation light, capturing an image of emitted light emitted from the object itself in response to the irradiation of the irradiation light, detecting the emitted light as a marker of the object based on the captured image, and performing a predetermined process based on the detected emitted light.

[0041] According to this aspect, emitted light is emitted from the object itself when irradiated with irradiating light, so that a specified area of ​​the object can be detected as a marker and a specified process can be performed without changing the appearance or shape of the object or impairing the mobility of the object.

[0042] A processing program according to another aspect of the present invention causes a computer to function as an irradiation unit that irradiates an object with irradiation light, an imaging unit that captures the emitted light emitted from the object itself in response to the irradiation of the irradiation light, a detection unit that detects the emitted light as a marker of the object based on the image captured by the imaging unit, and a processing unit that executes predetermined processing based on the emitted light detected by the detection unit.

[0043] According to this aspect, emitted light is emitted from the object itself when irradiated with irradiating light, so that a specified area of ​​the object can be detected as a marker and a specified process can be performed without changing the appearance or shape of the object or impairing the mobility of the object. [Effects of the Invention]

[0044] According to the present invention, a processing system, processing method, and processing program are provided that enable a specified area of ​​an object to be used as a marker without installing or applying anything that can be used as a marker to the object. [Brief explanation of the drawings]

[0045] [Figure 1] 1 is a diagram illustrating an example of the configuration of a processing system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing the physical configuration of the control device shown in FIG. [Figure 3] FIG. 3 is a diagram showing luminescence data acquired by the processing system according to the first embodiment of the present invention. [Figure 4] 1 is a diagram showing functional blocks of a processing system according to a first embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an image captured by an imaging device when an irradiation device emits pulsed irradiation light once. [Figure 6] 10A and 10B are diagrams illustrating an image captured by an imaging device when an irradiation device intermittently irradiates pulsed irradiation light multiple times. [Figure 7] 10A and 10B are diagrams showing an image captured by an imaging device when an irradiation device continuously irradiates irradiation light. [Figure 8A] 10A and 10B are diagrams showing an image captured by an imaging device immediately after a rotating object is irradiated with irradiation light and the irradiation light is stopped. [Figure 8B] FIG. 10 is a diagram showing an image captured by an imaging device 100 milliseconds after a rotating object is irradiated with irradiation light and the irradiation light is stopped. [Figure 9] 4 is a flowchart of a tracking process executed by the processing system according to the first embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing functional blocks of a processing system according to a first modified example of the first embodiment of the present invention. [Figure 11A]FIG. 10 is a diagram showing an image captured by an imaging device while a milk carton is being irradiated with spatially modulated irradiation light. [Figure 11B] FIG. 10 is a diagram showing an image captured by an imaging device immediately after spatially modulated irradiation light is irradiated onto a milk carton and the irradiation is stopped. [Figure 12A] FIG. 10 is a diagram showing an image captured by an imaging device while a sketchbook is irradiated with spatially modulated illumination light. [Figure 12B] FIG. 10 is a diagram showing an image captured by an imaging device immediately after spatially modulated illumination light is irradiated onto a sketchbook and stopped. [Figure 13] FIG. 10 is a diagram showing light emission data when the intensity of the irradiated light is modulated. [Figure 14A] FIG. 14 is a diagram showing a part of an image captured under conditions corresponding to the light emission data L1 of FIG. [Figure 14B] FIG. 14 is a diagram showing a part of an image captured under conditions corresponding to the light emission data L1 of FIG. [Figure 15A] FIG. 14 is a diagram showing a part of an image captured under conditions corresponding to the light emission data L2 in FIG. [Figure 15B] FIG. 14 is a diagram showing a part of an image captured under conditions corresponding to the light emission data L2 in FIG. [Figure 16A] FIG. 14 is a diagram showing a part of an image captured under conditions corresponding to light emission data L3 in FIG. [Figure 16B] FIG. 14 is a diagram showing a part of an image captured under conditions corresponding to light emission data L3 in FIG. [Figure 17] FIG. 10 is a diagram showing functional blocks of a processing system according to a second modified example of the first embodiment of the present invention. [Figure 18] FIG. 10 is a diagram illustrating an example of the configuration of a processing system according to a third modified example of the first embodiment of the present invention. [Figure 19] FIG. 10 is a diagram showing the overall configuration of a processing system according to a second embodiment of the present invention. [Figure 20] FIG. 10 is a diagram showing the overall configuration of a processing system according to a third embodiment of the present invention. [Figure 21A]21 is a diagram showing an image captured by an imaging device immediately after irradiating light onto granulated sugar outside an obstacle in the operating environment of the processing system shown in FIG. 20 and stopping the irradiation. FIG. [Figure 21B] FIG. 21 is a diagram showing an image captured by an imaging device 200 milliseconds after irradiating light onto granulated sugar outside an obstacle in the operating environment of the processing system shown in FIG. 20 and stopping the irradiation. [Figure 22A] 21 is a diagram showing an image captured by an imaging device immediately after irradiating light onto granulated sugar inside an obstacle in the operating environment of the processing system shown in FIG. 20 and stopping the irradiation. FIG. [Figure 22B] FIG. 21 is a diagram showing an image captured by an imaging device 200 milliseconds after light is irradiated onto granulated sugar inside an obstacle in the operating environment of the processing system shown in FIG. 20 and the light stops irradiating the granulated sugar. [Figure 23A] FIG. 10 is a diagram showing an image captured when the sketchbook is irradiated with light. [Figure 23B] FIG. 10 is a diagram showing an image captured while the sketchbook is moving. [Figure 23C] FIG. 10 is a diagram showing the trajectory of light emitted from the sketchbook. [Figure 23D] FIG. 10 is a diagram showing an image reconstructed by deconvolution. [Figure 24] FIG. 13 is a diagram showing the overall configuration of a processing system according to a seventh embodiment of the present invention. [Figure 25] FIG. 13 is a diagram showing the overall configuration of a processing system according to an eighth embodiment of the present invention. [Figure 26] FIG. 2 is an explanatory diagram for explaining types of emitted light. [Figure 27] FIG. 19 is a diagram showing the overall configuration of a processing system according to a tenth embodiment of the present invention. [Figure 28] 28 is a diagram illustrating an example of a temperature change when a laser is irradiated onto an object for different irradiation times in the operating environment of the processing system shown in FIG. 27. FIG. [Figure 29A] 10A and 10B are diagrams showing experimental results when a point cloud is drawn as an emission light marker. [Figure 29B] FIG. 10 is a diagram showing experimental results when the Chinese character "Hikari" (light) is drawn using an emitting light marker. DETAILED DESCRIPTION OF THE INVENTION

[0046] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to one aspect of the present invention will now be described with reference to the accompanying drawings. In the drawings, components with the same reference numerals have the same or similar configurations.

[0047] 1. First Embodiment 1 is a diagram showing an example of the configuration of a processing system according to the first embodiment. The processing system 10 is a system that can use emitted light from the object 100 itself as a marker, without attaching a marker to the object 100.

[0048] As shown in FIG. 1, the processing system 10 includes, for example, an irradiation device 11, an imaging device 12, and a control device 13.

[0049] The irradiation device 11 irradiates the object 100 with irradiation light under any irradiation conditions. The irradiation device 11 may be, for example, an ultraviolet laser or an ultraviolet LED (Light Emitting Diode). The wavelength of the irradiation light irradiated by the irradiation device 11 may be 10 nm or more. By using irradiation light with a wavelength of 10 nm or more, management when irradiating the object 100 with irradiation light can be simplified, and the operating costs of the processing system 10 can be reduced. The irradiation device 11 may also include one or more light sources that generate irradiation light of multiple different wavelengths. The irradiation device 11 may include a wavelength-tunable laser or LED as a light source, or may include multiple lasers or LEDs with different wavelengths as light sources. The irradiation light irradiated by the irradiation device 11 may be collimated light. The irradiation device 11 may irradiate, for example, pulsed irradiation light single-shot or intermittently, or may irradiate irradiation light continuously.

[0050] When the object 100 is irradiated with irradiation light, emitted light is emitted from the region of the object 100 irradiated with the irradiation light. In this specification, "emitted light" refers to light emitted from the object itself in response to irradiation with irradiation light, and is emitted with a delay compared to the light reflected by the object. Emitted light can be observed even after the irradiation of the object 100 is stopped. For example, when the object is irradiated with irradiation light, electrons in the molecules are excited by the absorption of the excitation light in the object, and energy is released when the electrons return to their ground state through various processes. Emitted light includes delayed fluorescence, phosphorescence, afterglow, or phosphorescence, which are generated by such electron excitation in the object. Alternatively, the emitted light may be emitted light (infrared) that is absorbed by the light irradiation (light energy, photoexcitation) of the object and converted into thermal energy (wavelength conversion). Note that stopping the irradiation of the object 100 with irradiation light is not limited to turning off the irradiation device 11, but also includes various configurations to prevent the irradiation light from hitting the object 100, such as changing the optical path of the irradiation light.

[0051] The imaging device 12 captures an image of the emitted light emitted from the object 100 in response to irradiation with the irradiating light. By capturing an image of the emitted light, the imaging device 12 can obtain changes in the position of the emitted light over time and changes in the emission intensity over time, and thus by using the emitted light as a marker, various measurements of the object 100 become possible. Hereinafter, the time during which the emitted light can be observed by the imaging device 12 is also referred to as the "emission duration."

[0052] The imaging device 12 may be, for example, a high-speed camera that captures the emitted light at a frame rate of 100 fps or more. By using a high-speed camera as the imaging device 12, it is possible to detect the emitted light even if the emission duration of the emitted light is several milliseconds to several tens of milliseconds. Furthermore, since the emitted light can be captured with a relatively short exposure time, it is possible to reduce the amount of movement of the object 100 between image frames and to reduce the occurrence of motion blur compared to when capturing images with a relatively long exposure time. The frame rate of the high-speed camera may be 1,000 fps, 10,000 fps, or more.

[0053] Furthermore, the imaging device 12 may be equipped with, for example, a wide-angle lens, a zoom lens, or a microscope lens (e.g., an objective lens, an eyepiece, etc.) depending on the distance to the object 100 and the size of the object 100. For example, if the object 100 is a large object located far from the imaging device 12, a wide-angle lens or a zoom lens may be equipped. For example, if the object 100 is a micron-order object (e.g., a microorganism, a bacterium, etc.) located close to the imaging device 12, a microscope lens or the like may be equipped. By appropriately selecting the lens of the imaging device 12, it is possible to enlarge the object 100 when it is small, or reduce it when it is large to obtain a wide angle of view. The distance from the imaging device 12 to the object 100 may be long or short.

[0054] Any light receiving sensor may be applied to the imaging device 12 depending on the wavelength spectrum of the light emitted from the object 100. The wavelength spectrum of the light emitted from the object 100 is, for example, from the ultraviolet region to the visible light region, but depending on the type of object 100, the wavelength of the irradiated light, etc., it may extend into the infrared region, so it is desirable to select an appropriate light receiving sensor.

[0055] The emitted light captured by the imaging device 12 includes emitted light generated during irradiation of the irradiation light and emitted light generated after irradiation of the irradiation light on the object 100 is stopped. By capturing an image of the emitted light generated after irradiation of the irradiation light is stopped, it is possible to easily distinguish between the irradiation light and the emitted light. Note that the method for distinguishing between the irradiation light and the emitted light is not limited to this. For example, if the emitted light is generated by excitation of electrons in the object 100, the wavelength spectra of the irradiation light and the emitted light will be different from each other. Therefore, for example, a filter that blocks light with the wavelength of the irradiation light and transmits light with the wavelength of the emitted light may be provided on the optical path of the imaging device 12 to prevent the irradiation light from directly entering the imaging device 12.

[0056] In this embodiment, it is assumed that emitted light is emitted from the object 100 toward the irradiation device 11, and an example is shown in which the image capturing device 12 is provided on the side of the irradiation device 11. On the other hand, if the object 100 is made of a light-transmitting material and the emitted light passes through the object 100, causing the intensity of the emitted light to be greater on the transmitted light side than on the irradiation device 11 side, the image capturing device 12 may be disposed on the transmitted light side.

[0057] The control device 13 controls the irradiation by the irradiation device 11 by sending a signal to the irradiation device 11, and controls the imaging by the imaging device 12 by sending and receiving signals to and from the imaging device 12. The control device 13 also detects emitted light based on the image captured by the imaging device 12. When the control device 13 detects the emitted light, the emitted light emitted from the object 100 itself functions as a marker of the object 100. In this specification, making a predetermined area of ​​the object itself function as a marker without attaching or applying a physical marker or a substance that functions as a marker to the object is also referred to as "marking."

[0058] FIG. 2 is a diagram showing the physical configuration of the control device shown in FIG. 1. The control device 13 has a CPU (Central Processing Unit) 20 corresponding to a calculation unit, a RAM (Random Access Memory) 21 and a ROM (Read Only Memory) 22 corresponding to a storage unit, a communication unit 23, an input unit 24, and a display unit 25. These components are connected via a bus so that they can send and receive data to each other. In this example, the control device 13 is configured by one computer, but the control device 13 may also be realized by combining multiple computers. The configuration shown in FIG. 2 is an example, and the control device 13 may have other components or may not have some of these components.

[0059] The CPU 20 is a control unit that controls the execution of programs stored in the RAM 21 or the ROM 22 and performs data calculations and processing. The CPU 20 is a calculation unit that executes a program (hereinafter also referred to as a "marking program") that irradiates an object with irradiation light and detects light emitted from the object itself. The CPU 20 receives various data from the communication unit 23 and the input unit 24, and displays the results of calculations on the data on the display unit 25 or stores the results in the RAM 21 or the ROM 22.

[0060] The RAM 21 is a memory unit in which data can be rewritten, and may be configured, for example, with a semiconductor memory element. The RAM 21 may store a marking program executed by the CPU 20. Note that these are merely examples, and the RAM 21 may store data other than these, or may not store some of these.

[0061] The ROM 22 is a storage unit from which data can be read, and may be configured, for example, with a semiconductor memory element. The ROM 22 may store, for example, a marking program or data that is not rewritten.

[0062] The communication unit 23 is an interface that connects the control device 13 to other devices, and may be connected to a communication network such as the Internet.

[0063] The input unit 24 receives data input from a user, and may include, for example, a keyboard and a touch panel.

[0064] The display unit 25 visually displays the results of calculations performed by the CPU 20, and may be configured with, for example, an LCD (Liquid Crystal Display). The display unit 25 may display images captured by the imaging device 12, etc.

[0065] The marking program may be provided by being stored in a computer-readable storage medium such as RAM 21 or ROM 22, or may be provided via a communication network connected by communication unit 23. In control device 13, CPU 20 executes the marking program to realize various operations described below. Note that these physical configurations are merely examples and do not necessarily have to be independent configurations. For example, control device 13 may include an LSI (Large-Scale Integration) in which CPU 20 is integrated with RAM 21 and ROM 22.

[0066] Returning to FIG. 1 , the object 100 may be any object. However, the object 100 in this embodiment is not one on which a phosphorescent material is artificially applied, but rather an object that emits light for several milliseconds to several hundred milliseconds or longer due to a phosphorescent phenomenon based on the electronic structure of the object itself. The object 100 can be broadly classified into artificial objects and non-artificial objects. Examples of artificial objects include, but are not limited to, paper such as milk cartons, sketchbooks, and office paper; structures such as concrete; foods such as granulated sugar, sugar, and chocolate; and fabrics such as cloth and textiles. Examples of non-artificial objects include, but are not limited to, ores, wood, and living organisms (such as animals and plants, microorganisms, and bacteria). By using light emitted by the phosphorescent phenomenon as a marker, the emitted light can be detected robustly regardless of the appearance and shape of the object 100.

[0067] The object 100 may move relative to at least a part of the processing system 10. For example, FIG. 1 shows a state in which the object 100 is stationary, and the illumination device 11, the imaging device 12, and the control device 13 move together in the direction of the arrow. By detecting the emitted light as a marker, the processing system 10 can measure, for example, the shape of the object 100, and the relative position, orientation, and velocity of the object 100 with respect to the processing system 10. The processing system 10 can calculate the position, orientation, and velocity of the object 100 in real space based on the measured relative position, orientation, and velocity.

[0068] FIG. 3 is a diagram showing light emission data acquired by the processing system 10 according to this embodiment. The light emission data L shown in the figure is obtained by irradiating a sketchbook, which is an example of the object 100, with a laser having a wavelength of 375 nm as irradiation light for 500 milliseconds, capturing an image at 120 fps using the imaging device 12 after the irradiation has stopped, and expressing the pixel value of the pixel at the center coordinate of the area irradiated with the irradiation light from the pixels in the captured image in 10 bits (0 to 1023). In FIG. 3, the horizontal axis represents the number of image frames, and the vertical axis represents the pixel value in the area irradiated with the irradiation light. The pixel value corresponds to the emitted light intensity (light emission intensity).

[0069] According to the light emission data L, the pixel value immediately after the irradiation of the light is stopped is relatively high, that is, a strong light emission is observed, but it can be seen that the pixel value gradually attenuates from there.

[0070] The light emission data L after irradiation of the irradiation light is stopped is divided into an exponential decay section B1 and a gradual decay section B2. When irradiation of the irradiation light is stopped, the exponential decay section B1 begins immediately, and pixel values ​​exponentially decay. In this example, the exponential decay section B1 continues for about 9 frames (75 milliseconds) after imaging begins, during which the pixel values ​​decay by about 80% from their maximum value. Following the exponential decay section B1, a gradual decay section B2 continues for about 21 frames (175 milliseconds), during which the pixel values ​​decay to their minimum value.

[0071] Therefore, if the attenuation characteristics of the pixel values ​​of emitted light from a certain object are known, it is possible to estimate the elapsed time since the irradiation of the irradiated light was stopped based on the pixel values ​​of the emitted light detected at a certain time. In other words, it can be said that a marker made of emitted light contains both spatial information indicating its position and temporal information indicating the elapsed time.

[0072] In this example, an image after irradiation with irradiation light is captured by one imaging device 12 at 120 fps, but the imaging device 12 may include a camera that captures the emitted light of the object 100 at a frame rate lower than 120 fps and a high-speed camera, and the emitted light in a period when time changes relatively quickly may be captured by the high-speed camera, and the emitted light in a period when time changes relatively slowly may be captured by the camera. Here, the period when time changes relatively quickly may be, for example, the exponential decay interval B1, and the period when time changes relatively slowly may be, for example, the decay interval B2. In this case, the high-speed camera may be one that can capture changes in brightness on the order of 10 nanoseconds, i.e., 10 8 In this way, by using two cameras, one high-speed camera and one relatively slow-speed camera, it is possible to capture continuous images with sufficient time resolution both in periods when the time change of the emitted light is relatively fast and in periods when it is relatively slow.

[0073] In this example, the irradiation time of the irradiation light is set to 500 milliseconds, but the irradiation time is not limited to this. The irradiation conditions, such as the wavelength, irradiation intensity, and irradiation time of the irradiation light, basically depend on the electronic structure of the object 100. For example, if the object 100 is a sketchbook, the lifetime of the emitted light tends to saturate if the irradiation time of the irradiation light is set to about 700 milliseconds or more. In this way, the irradiation time of the irradiation light on the object 100 may be set to be equal to or longer than the irradiation time at which the lifetime of the emitted light saturates.

[0074] The processing system 10 can be applied to various applications, but in this embodiment, a case where it is used for tracking an object will be described. FIG. 4 is a diagram showing functional blocks of the processing system according to this embodiment. The irradiation device 11 includes an irradiation unit 110. The imaging device 12 includes an imaging unit 120. The control device 13 includes a setting unit 130, a detection unit 131, and a processing unit 132.

[0075] The irradiation unit 110 irradiates the object 100 with irradiation light based on the control of the control device 13. The imaging unit 120 captures an image of the emitted light emitted from the object 100 based on the control of the control device 13. The captured image is transmitted to the control device 13. Note that, in the present embodiment, an example is shown in which the irradiation device 11 includes the irradiation unit 110 and the imaging device 12 includes the imaging unit 120, but these devices do not necessarily have to be independent configurations. For example, the control device 13 may have the functions of the irradiation unit and the imaging unit.

[0076] The setting unit 130 sets the irradiation conditions of the irradiation light emitted by the irradiation device 11, and also sets the imaging conditions of the imaging device 12. The irradiation conditions of the irradiation light include at least one of the wavelength of the irradiation light, the intensity of the irradiation light, the irradiation time of the irradiation light, and the interval of the irradiation light. The wavelength of the irradiation light may be, for example, 200 nm to 400 nm in the ultraviolet region, but may also be 200 nm or less in the far ultraviolet region, or 400 nm or more in the visible light region.

[0077] The shorter the wavelength of the irradiated light, the stronger the intensity of the irradiated light, and the longer the irradiation time of the irradiated light, the longer the light emission duration of the emitted light tends to be. Therefore, in order to track an object, the wavelength of the irradiated light may be set to be relatively short, the intensity of the irradiated light to be relatively strong, and the irradiation time of the irradiated light to be relatively long. However, since the light emission duration of the emitted light may not change even if the irradiation time of the irradiated light is extended beyond a certain time, the irradiation time of the irradiated light may be set to the shortest time that allows the light emission duration of the emitted light to approach its maximum value.

[0078] The imaging conditions of the imaging device 12 include at least one of a frame rate, a frame period, an exposure time, and a resolution. Note that the irradiation conditions of the irradiating light may be appropriately designed according to the frame rate and exposure time of the imaging device 12, for example.

[0079] The detection unit 131 captures an image captured by the imaging device 12, and based on the image, detects emitted light emitted from the object 100. The method of detecting the emitted light is not particularly limited, but may include, for example, at least one of binarizing the captured image, improving the S / N ratio by performing expansion or contraction processing, selecting the region with the largest outline from among the extracted multiple regions, and determining the coordinates of the center of gravity of the selected region.

[0080] The processing unit 132 executes predetermined processing based on the position coordinates in the image of the emitted light detected by the detection unit 131. In this embodiment, the processing unit 132 tracks the object 100 by measuring, for example, the shape of the object 100 and at least one of the relative position, relative orientation, and relative velocity of the object 100 with respect to the processing system 10. The processing unit 132 calculates the position, orientation, and velocity of the object 100 in real space based on the measured relative position, relative orientation, and relative velocity. As the specific tracking method can be the same as a configuration using physical markers or feature points of the object as markers, a detailed description thereof will be omitted.

[0081] With the above configuration, the processing system 10 can track the object 100, for example, based on the measured position of the object in real space. Tracking the object 100 includes tracking the movement of the object 100 within an image and varying the field of view of the imaging device in response to the movement of the object 100 so that the object 100 remains within the field of view of the imaging device.

[0082] Next, a specific method for tracking an object under various illumination conditions and imaging conditions will be described with reference to Figures 5 to 7. For convenience, the following description will be given taking as an example a case where the illumination device 11 emits point-like illumination light, but the illumination light is not limited to point-like illumination light.

[0083] 5 is a diagram showing an image captured by the imaging device when the irradiation device emits a single pulse of irradiation light. In this embodiment (hereinafter also referred to as "embodiment A"), the light emission duration of the emitted light is assumed to be longer than the exposure time of the imaging device. In other words, the next image is assumed to be captured within the light emission duration of the emitted light captured in a certain image. The images shown in (a) to (d) in FIG. 5 are assumed to be arranged in the order in which they were captured by the imaging device 12.

[0084] When the irradiation device 11 irradiates a pulse of irradiation light once, emitted light is emitted from the area of ​​the object 100 that is irradiated with the irradiation light. In other words, a predetermined area of ​​the object 100 is marked. Thereafter, even if the object 100 moves relative to the imaging device 12, emitted light continues to be emitted from the same area for the light emission duration.

[0085] The imaging device 12 captures images of the object 100 at multiple times. The detection unit 131 detects the emitted light from each captured image. At this time, the object 100 moves relative to the imaging device 12, and therefore, as shown in FIG. 5, the position of the detected emitted light gradually shifts between images. Therefore, the processing unit 132 can calculate the relative velocity of the object 100 based on the shift in the position of the emitted light in the multiple captured images. Note that FIG. 5 is an illustration assuming that the exposure time of the imaging device 12 is extremely short compared to the relative velocity of the object 100. In reality, if the object 100 moves during the exposure time of the imaging device 12, each emitted light will have a predetermined length, as shown in FIG. 7, which will be described later. This also applies to FIG. 6, which will be described later.

[0086] 5, if the positions of emitted light detected in multiple images are spaced apart by X (pixels) and the frame period of the image capture device 12 is t (seconds), the pixel relative velocity Vp of the object 100 is calculated as Vp=X / t (pixels / second). For example, if the surface of the object 100 facing the image capture device 12 is flat and the object 100 moves translationally relative to the image capture device 12, the pixel relative velocity Vp can be converted into an actual relative velocity in real space.

[0087] In aspect A, the higher the frame rate of the imaging device 12, the narrower the pixel interval of the emitted light detected between each image, thereby improving the accuracy of the calculated relative velocity of the object 100. Furthermore, in aspect A, the object 100 can be tracked based on the transition of the position of the emitted light emitted in response to a certain irradiation, so the relative velocity of the object 100 can be calculated regardless of the light emission characteristics of the emitted light or the distance between the irradiation device 11 and the imaging device 12. Note that, although the above description has been given as an example of a case where the irradiation device 11 irradiates the irradiation light once, a similar method can also be applied to a case where the irradiation device 11 irradiates the irradiation light multiple times.

[0088] 6 is a diagram showing an image captured by an imaging device when an irradiation device intermittently irradiates pulsed irradiation light multiple times. In this embodiment (hereinafter also referred to as "embodiment B"), the light emission duration of the emitted light is longer than the interval between irradiation lights. In other words, the next irradiation light is irradiated within the light emission duration of the emitted light emitted in response to a certain irradiation.

[0089] Since the object 100 is moving relative to the irradiation device 11, when the irradiation device 11 intermittently irradiates pulsed irradiation light in the same direction for multiple timings (four times in FIG. 6), emitted light is emitted from multiple different regions of the object 100. That is, multiple regions of the object 100 (four in FIG. 6) are marked.

[0090] The imaging device 12 captures an image of the object 100, which moves relative to the imaging device 12, at least once. The detection unit 131 detects emitted light from the captured image. Because the object 100 is marked with multiple regions, multiple emitted lights are detected from a single image. Furthermore, the intensity of the emitted light attenuates depending on the luminescence characteristics of the material of the object 100. As shown in FIG. 6, the intensity of the emitted light detected by the detection unit 131 gradually increases from the emitted light corresponding to the most recent irradiation to the emitted light corresponding to the most recent irradiation. Therefore, the processing unit 132 can identify the multiple emitted lights detected in a single image in chronological order based on their intensities. This allows the processing unit 132 to calculate the relative velocity of the object 100 with respect to the imaging device 12 based on the pixel spacing between multiple emitted lights whose intensities are adjacent to each other.

[0091] 6, when the pixel interval between two emitted lights whose intensities are adjacent to each other and detected on one image is X (pixels) and the irradiation interval by the irradiation device 11 is z (seconds), the pixel relative velocity Vp of the object 100 is calculated by Vp=X / z (pixels / second). In this case, as in the above-described embodiment A, when the surface of the object 100 facing the image capture device 12 is flat and the object 100 moves translationally relative to the image capture device 12, the pixel relative velocity Vp can be converted into the real relative velocity in real space.

[0092] In aspect B, the shorter the interval of the irradiation device 11, the narrower the pixel interval of the emitted light detected in a certain image, and therefore the accuracy of the calculated relative velocity of the object 100 improves. Also, in aspect B, as in aspect A, the relative velocity of the object 100 can be calculated regardless of the distance between the irradiation device 11 and the image capture device 12. Aspects A and B are effective, for example, when the object 100 does not deform or when it is sufficient to track the movement of a single point on the object 100. In such cases, it is preferable that the irradiation time of the irradiation light is short. Note that in aspect B, when the direction of movement of the object 100 is known, such as when the object 100 moves relatively in a linear manner, the relative velocity of the object 100 may be calculated based on multiple spatially adjacent emitted lights without using the light emission characteristics of the emitted light.

[0093] 7 is a diagram showing an image captured by an imaging device when the irradiation device continuously irradiates irradiation light. In this embodiment (hereinafter also referred to as "embodiment C"), the irradiation device 11 irradiates once for a predetermined time and then stops, but instead, the irradiation device 11 may irradiate irradiation light for a predetermined time multiple times, or may continue to irradiate irradiation light without stopping.

[0094] Since the object 100 is moving relative to the irradiation device 11, when the irradiation device 11 continuously irradiates irradiation light, emitted light is emitted from an area having a predetermined length on the object 100. That is, in embodiment C, the object 100 is marked linearly.

[0095] The imaging device 12 captures an image of the object 100 at least once. The detection unit 131 detects emitted light from the captured image. The trajectory of the detected emitted light indicates the trajectory of the relative movement of the object 100. Here, since the intensity of the emitted light attenuates over time, it can be seen that the stronger the intensity of the detected emitted light, the more recently irradiated the area is, and the weaker the intensity of the emitted light, the more recently irradiated the area is. Therefore, the processing unit 132 can measure the movement trajectory of the object 100 relative to the irradiation device 11 based on the trajectory of the emitted light detected from one image and the intensity of the emitted light. In aspect C, the longer the irradiation time of the irradiation light, the larger the area of ​​the trajectory of the emitted light is, and therefore the amount of information obtained from the trajectory of the object 100 per image increases.

[0096] Furthermore, the emitted light is emitted with luminous characteristics according to the material of the object 100. Therefore, the control device 13 may store in advance the luminous characteristics of the emitted light of the object 100 or the same material as the object 100, and calculate the relative speed of the object 100 based on a comparison between the stored luminous characteristics and the detected luminous characteristics of the emitted light of the object 100. The luminous characteristics of the emitted light include, for example, attenuation data indicating the relationship between the elapsed time from irradiation of the irradiating light and the luminous intensity of the emitted light, the luminous duration of the emitted light, and the rate of change of the luminous intensity of the emitted light.

[0097] For example, if the light emission duration τ (seconds) of emitted light at a certain irradiation light intensity is stored in advance and the length of the trajectory of the detected emitted light is X (pixels), the pixel relative velocity Vp of the object can be calculated by Vp = X / τ (pixels / second). In aspect C, compared to aspects A and B described above in which pulsed irradiation light is emitted, a more precise and continuous relative velocity vector can be calculated without being restricted by the frame rate of the image capture device 12 or the irradiation rate of the irradiation device 11. Note that in aspect C, if the direction of movement of the object 100 is known and the light emission duration of the emitted light is longer than the exposure time of the image capture device, the relative velocity of the object 100 may be calculated based on the length of the trajectory of the emitted light without using the light emission characteristics of the emitted light.

[0098] FIG. 8A shows an image captured by an imaging device immediately after irradiating a rotating object with irradiation light and stopping the irradiation light. FIG. 8B shows an image captured by an imaging device 100 milliseconds after irradiating a rotating object with irradiation light and stopping the irradiation light. Specifically, the images shown in FIGS. 8A and 8B are captured when the irradiation device 11 and imaging device 12 are stationary and a chocolate object, an example of an object, is placed on a turntable rotating at an angular velocity of 17.5 rad / s on a plane parallel to the imaging surface of the imaging device 12. In this experiment, the wavelength of the irradiation light emitted by the irradiation device 11 is 375 nm, and the irradiation time is 500 milliseconds. The imaging device 12 is a CMOS camera combined with an image intensifier with a gain set to 5.0. Hereinafter, unless otherwise specified, the wavelength of the irradiation light is assumed to be 375 nm. The resolution of the imaging device 12 is 640 × 480 pixels, and the imaging frame rate is 240 fps. 8A and 8B are images obtained after the processing device 13 has performed binarization processing on the original image captured by the imaging device 12 using a predetermined threshold value.

[0099] As shown in Figures 8A and 8B, the trajectory of emitted light 200 emitted from the chocolate describes an arc, and its intensity decreases counterclockwise. As shown in Figure 18, which will be shown later, in this experiment, the chocolate was imaged through a mirror, and the captured image was flipped left to right. Therefore, from the trajectory of emitted light 200, it can be seen that the chocolate is rotating clockwise. If the emission duration of the emitted light from the chocolate is τ (seconds) and the rotation angle of the trajectory of the emitted light is Y (rad), the relative angular velocity ω of the chocolate can be calculated as ω = Y / τ (rad / second).

[0100] This method of analyzing the trajectory of emitted light based on a single image captured at a given time can be used, for example, during the emission duration, when the emitted light from the area irradiated at the start of laser irradiation remains observable even after attenuation. On the other hand, if the laser irradiation time is longer than the emission duration, the bright end points of the emission light trajectory in two images captured at different times, i.e., the points corresponding to the area most recently irradiated by the laser (just before the end of laser irradiation), can be detected. The relative angular velocity ω can be calculated by dividing the difference in angle (rad) between the two detected points as seen from the center of rotation of the turntable by the inter-frame time (seconds). For example, in this experiment, the time between the frames in Figures 8A and 8B was 100 milliseconds, and the difference in angle between the bright end points in each image was 1.77 rad. Therefore, the relative angular velocity ω was calculated as ω = 1.77 / 0.1 = 17.7 rad / s. This indicates that the angular velocity of the turntable with the chocolate loaded, 17.5 rad / s, was calculated fairly accurately.

[0101] FIG. 9 is a flowchart of the tracking process executed by the processing system according to this embodiment.

[0102] First, the setting unit 130 sets the irradiation conditions of the irradiation device 11 (step S10) and also sets the imaging conditions of the imaging device 12 (step S11). Next, the irradiation device 11 irradiates the object 100 with irradiation light under the set irradiation conditions (step S12).

[0103] Next, the imaging device 12 captures the emitted light emitted from the object 100 itself in response to the irradiation of the irradiating light under the set imaging conditions (step S13). Next, the detection unit 131 detects the emitted light based on the captured image (step S14). Next, the processing unit 132 calculates the relative speed of the object 100 based on the transition or trajectory of the detected emitted light (step S15). This completes the tracking process.

[0104] As described above, various image processing may be performed on the captured image in step S14. For example, when detecting emitted light, if the captured image contains fixed noise or the like that occurs during shooting, the emitted light may be detected after removing this noise. For example, an image may be captured separately after a sufficient amount of time has passed since the emitted light has decayed, and the maximum pixel value may be taken to evaluate the magnitude of the pixel values ​​of the fixed noise, and a binary image may be created using this evaluation value as a threshold. Since the fixed noise is removed from the created binary image, the center of gravity of the bright area on the image resulting from the emitted light can be considered to be the center of the emitted light.

[0105] As described above, in the processing system 10 according to this embodiment, emitted light from a predetermined region of the object itself can function as a marker, without attaching a physical marker to the object. This marking method allows measurement of the object without changing the appearance or shape of the object or impairing the object's mobility, as compared to configurations in which physical markers are attached to the object or a phosphorescent material is applied to the object.

[0106] The processing system 10 according to this embodiment can function as a marker even if the environment in which the object 100 exists is the atmosphere, a vacuum, or a liquid. Furthermore, even if the object 100 is exposed to an electric field or a magnetic field, it is not affected by these.

[0107] The processing system 10 according to this embodiment can robustly attach an emitted light marker to an object regardless of its color, material, shape, etc. This allows even a plain white object to be measured with high accuracy.

[0108] In the processing system 10 according to this embodiment, emitted light that is emitted for several milliseconds to several hundred milliseconds based on the electronic structure of the material that constitutes the object is used as a marker. Therefore, compared to the case where a laser with a relatively high light intensity is used, marking can be performed without destroying the object or leaving traces in the irradiated area of ​​the object due to photo-alteration.

[0109] For example, in a configuration in which a projector projects a pattern of light onto an object, the position of the pattern of light relative to the object shifts when the object moves, and therefore the pattern of light cannot be used as a marker for the object. On the other hand, in the processing system 10 according to this embodiment, once marking is completed, even if the object 100 moves thereafter, the emitted light continues to be emitted from the same region for the duration of the light emission, and therefore this can be applied to tracking of the object, etc.

[0110] In the configuration described in Patent Document 2, the speed of the conveyor belt is calculated based on the relationship between the timing of light emission by the light emitter and the timing of light reception by the light receiver, and the distance between the light emitter and the light receiver. On the other hand, in the processing system 10 according to the present embodiment, the relative speed of the object with respect to the imaging device can be calculated based on the transition of the position of the emitted light, regardless of the distance between the irradiation device and the imaging device. Therefore, compared to the configuration described in Patent Document 2, the processing system 10 has a greater degree of freedom in the movement of the measurable object, making it possible to measure, for example, the angular velocity of the object.

[0111] In the above-described embodiment, an example of an application of the processing system 10 has been described in which the processing unit 132 tracks an object, but the processing system according to this embodiment may also be used for various other purposes.

[0112] 10 is a diagram showing functional blocks of a processing system according to a first modification of this embodiment. Note that from this modification onwards, descriptions of matters common to the above-mentioned embodiment will be omitted, and only differences will be described. In particular, similar effects resulting from similar configurations will not be mentioned in each embodiment and modification.

[0113] The processing system 10A according to this modification is different from the above-described processing system 10 in that the control device 13A further includes a modulation unit 133. For example, when multiple processing systems simultaneously perform marking, each processing system includes a modulation unit 133 that modulates the irradiated light in order to identify which processing system the irradiated light originates from. Note that, although FIG. 10 shows an example in which the control device 13A constitutes the modulation unit 133, the components constituting the modulation unit are not limited to the control device 13A and may be, for example, an irradiation device, or may be components provided separately from the irradiation device, imaging device, and control device.

[0114] The modulation unit 133 may spatially modulate the irradiated light, for example, by changing the graphic pattern of the irradiated light spread on a two-dimensional plane. If the irradiated light has a graphic pattern other than a dot, the emitted light also takes on a graphic pattern corresponding to the irradiated light, making it possible to identify the type of emitted light used as a marker based on the graphic pattern of the emitted light. For example, the processing system 10A may further include a diffraction grating provided at the irradiation port of the irradiation device 11 to generate linear irradiated light tilted in various directions. In this case, the modulation unit may include a diffraction grating.

[0115] 11A and 11B are diagrams showing images captured by an imaging device when spatially modulated irradiation light is irradiated onto a milk carton. FIGS. 12A and 12B are diagrams showing images captured by an imaging device when spatially modulated irradiation light is irradiated onto a sketchbook. FIGS. 11A and 12A show images captured during irradiation of the irradiation light, and FIGS. 11B and 12B show images captured immediately after the irradiation light stopped. The frame rate of the imaging device is 120 fps, and the images are taken when the irradiation light is linear.

[0116] As shown in Figures 11B and 12B, when a linear illumination light is irradiated, a linear emission light is captured even after the illumination light is stopped. These emission lights extend in different directions, and by detecting the direction of the emission light, i.e., the orientation of the graphic pattern of the emission light, it is possible to identify which illumination light the emission light originates from. Furthermore, if the graphic pattern of the illumination light is asymmetric, the direction of the object may be identified from the orientation of the graphic pattern of the emission light.

[0117] Furthermore, the modulation unit 133 may dynamically spatially modulate the irradiated light instead of statically. For example, the modulation unit 133 may generate a graphic pattern of emitted light by irradiating a point-like irradiated light while moving it in a graphic shape within the light emission duration of the emitted light. The movement of the irradiated light may be achieved using, for example, a one-axis or two-axis galvanometer mirror whose operation is controlled by the control device 13A. Graphic patterns include, for example, linear, rectangular, circular, and polygonal shapes. By emitting emitted light of different graphic patterns, each emitted light can be distinguished. Note that the dynamic spatial modulation of the irradiated light may be achieved using, for example, a DMD (Digital Mirror Device) or LCoS (Liquid Crystal on Silicon) device.

[0118] Because emitted light gradually weakens over time, generating a geometric pattern of emitted light requires completing the geometric movement of the irradiated light within the emission duration of the emitted light corresponding to the initial irradiation. For example, consider the case where the irradiating device 11 emits continuous irradiated light to generate a linear pattern of length q, as shown in FIG. 7 . When a circular irradiated light of radius r is irradiated for time t, the emission duration of the emitted light is defined as mt, and the moving speed of the irradiated light is defined as v. For any point on this linear pattern to be illuminated by the irradiated light of radius r for time t, the moving speed v of the irradiated light must satisfy v≦2r / t (Equation 1). Furthermore, to irradiate length q within the emission duration mt at moving speed v, moving speed v must satisfy q+2r≦vmt (Equation 2). From Equations 1 and 2, the processing system 10A can generate a linear pattern of length q that satisfies q≦2(m−1)r.

[0119] Alternatively, the modulation unit 133 may intensity-modulate the irradiated light by changing the irradiation intensity of the light. For example, the processing system 10A may further include an ND (Neutral Density) filter provided at the irradiation port of the irradiation device 11, and the irradiated light intensity may be changed according to the light transmittance of the ND filter. Some ND filters have the property that the light transmittance can be changed by rotating them, and the optical density increases linearly (i.e., the transmittance decreases) as the rotation angle increases. Therefore, the modulation unit 133 may change the intensity of the irradiated light by, for example, controlling the rotation angle of such an ND filter to change the light transmittance. The ND filter may be either an absorptive or reflective type.

[0120] FIG. 13 shows light emission data obtained when the intensity of the irradiated light is modulated. The light emission data L1 to L3 shown in the figure represent pixel values ​​of pixels at the center coordinates of the area irradiated with the irradiated light in the image captured by the imaging device 12 after irradiating a milk carton with the irradiated light for 100 milliseconds. The light emission data L1 represents the result when no ND filter is used, the light emission data L2 represents the result when the ND filter is rotated 90 degrees (i.e., transmittance is approximately 30%), and the light emission data L3 represents the result when the ND filter is rotated 180 degrees (i.e., transmittance is approximately several percent). The imaging device 12 is a CMOS camera combined with an image intensifier with a gain set to 5.0. The 10th and 11th frames correspond to the time during which the irradiated light is irradiated, so no imaging is performed. The 11th and 12th frames and beyond correspond to the pixel values ​​of the emitted light.

[0121] 14A and 14B are diagrams showing a portion of an image captured under conditions corresponding to light emission data L1 in Fig. 13. Figs. 15A and 15B are diagrams showing a portion of an image captured under conditions corresponding to light emission data L2 in Fig. 13. Figs. 16A and 16B are diagrams showing a portion of an image captured under conditions corresponding to light emission data L3 in Fig. 13. Figs. 14A, 15A, and 16A each show an image captured immediately after the illumination light was stopped (the 11th frame in Fig. 13), and Figs. 14B, 15B, and 16B each show an image captured 100 milliseconds after the illumination light was stopped (the 22nd frame in Fig. 13).

[0122] 13, it can be seen that the stronger the intensity of the irradiated light, the higher the pixel value at the same time. This indicates that emitted light can be distinguished by modulating the intensity of the irradiated light.

[0123] In addition to the above modulation methods, the modulator 133 may modulate the irradiated light using other modulation methods, such as time modulation, which changes the irradiation period of the irradiated light; wavelength modulation, which changes the spectrum of the irradiated light; or pulse code modulation, which modulates the irradiated light into a pulsed form. In the case of pulse code modulation, it is preferable to drive pulses taking into account the decay time of the emitted light in order to ensure the contrast ratio of the emitted light to be captured. For example, it is preferable to set a condition such that the next irradiation of the irradiated light is performed when the level of the emitted light has decreased from its maximum value to a predetermined percentage, based on a decay curve of the emitted light obtained in advance.

[0124] In any of these modulation methods, irradiation light modulated differently from each other is irradiated, and the imaging unit captures the emitted light based on the modulated irradiation light, thereby making it possible to identify which processing system emitted the emitted light detected by the detection unit. Note that each processing system may be provided with a filter in a stage preceding the imaging unit that selectively passes the emitted light based on the modulated irradiation light. Furthermore, each modulation method may be applied in combination, such as spatially and temporally modulated irradiation light.

[0125] Furthermore, when an irradiation device in one processing system applies emission light markers multiple times, the irradiation light may be modulated differently from each other for multiple times. In this case, even if the imaging unit simultaneously captures multiple images of emission light, it is possible to identify when the detected emission light is based on the irradiation light that was irradiated.

[0126] FIG. 17 is a diagram showing functional blocks of a processing system according to a second modified example of this embodiment.

[0127] The processing system 10B of this modified example differs from the above-described processing system 10 in that the irradiation device 11 irradiates the target object 100 with irradiation light that spreads on a two-dimensional plane, and the control device 13B further includes a matching unit 134.

[0128] The irradiation light emitted by the irradiation device 11 is a random pattern light that spreads on a two-dimensional plane facing the object 100. When such irradiation light is emitted, emitted light is emitted from random pattern regions on the object 100. That is, in the processing system 10A, the object 100 is marked in a random pattern.

[0129] The imaging device 12 captures, at multiple timings, images of the randomly patterned emitted light emitted from the target object 100. The imaging device 12 has an angle of view that allows it to capture at least a portion of the randomly patterned emitted light.

[0130] The matching unit 134 performs pattern matching using a plurality of images captured by the imaging device 12. This enables the processing system 10B to measure the relative position, relative orientation, etc. of the object in the image.

[0131] As previously known, it can be difficult to apply a random pattern marker using artificial methods such as attaching a physical marker to an object or applying a phosphorescent material. In this regard, the processing system 10B can emit emitted light according to the pattern of the irradiated light, and by irradiating the irradiated light in a random pattern, the emitted light can be used as a random pattern marker. By performing pattern matching using a random pattern marker, pattern matching can be performed with higher accuracy than, for example, a configuration using a pattern marker with a predetermined rule. Note that the pattern of the irradiated light is not limited to a random pattern, and the irradiated light may have various other patterns. Furthermore, the shape of the area of ​​the object 100 irradiated with the pattern light may be a two-dimensional plane or a three-dimensional shape. In the case of a three-dimensional shape, if the surface shape is known, the same measurement as described above can be performed by detecting the pattern of emitted light generated by the combination of the pattern light and the surface shape of the object 100.

[0132] FIG. 18 is a diagram showing an example of the configuration of a processing system according to a third modified example of this embodiment.

[0133] 18, a processing system 10C further includes mirrors 14 and 15 compared to the processing system 10. Note that the control device 13 is not shown in FIG.

[0134] The mirror 14 is a specific example of an optical path control unit that guides the light emitted from the object 100 to the imaging device 12. The mirror 15 is a specific example of an optical path control unit that guides the illumination light emitted from the illumination device 11 to any region of the object 100.

[0135] The mirrors 14 and 15 may be, for example, fixed mirrors or galvanometer mirrors, such as one-axis or two-axis mirrors, whose operation is controlled by the control device 13. FIG. 18 shows an example in which the mirror 14 is a fixed mirror and the mirror 15 is a galvanometer mirror. In the processing system 10C, by using a galvanometer mirror as the mirror 15, the irradiation position of the irradiation light can be controlled more quickly than by moving the irradiation device 11 itself. Therefore, even when the object 100 moves relative to the irradiation device 11, it becomes easier to continuously irradiate a specific region of the object 100 with the irradiation light. Furthermore, even when the object is small in size or when there are restrictions on irradiation on the surface of the object, it becomes easier to attach an emission light marker to the object.

[0136] Using the processing system 10C, for example, when the irradiation device 11 intermittently irradiates irradiation light at intervals shorter than the emission duration of the emitted light, new irradiation light may be irradiated so as to overlap the emitted light corresponding to the previous irradiation before the emission light corresponding to the previous irradiation disappears. Specifically, the processing unit 132 in the control device 13 calculates the relative position and relative velocity of the emitted light emitted from the object 100, and calculates the position to which the next irradiation light should be irradiated based on the calculated relative position and relative velocity of the emitted light. The control device 13 controls the mirror 15 so that the irradiation light is irradiated at the calculated position. This allows the new emitted light to be superimposed on the emitted light corresponding to the previous irradiation, allowing for the emission of emitted light from a predetermined region of the object without interruption in time. In other words, the emission duration of the emitted light used as a marker can be made longer and brighter.

[0137] In addition, when calculating the position where the next irradiation light should be applied, if the irradiation position deviates from the previous irradiation position due to delays in each process such as the imaging process by the imaging device 12, the image processing process by the control device 13, the control process of the mirror 15, and the irradiation process of the irradiation light, the control device 13 may calculate the position where the irradiation should be applied taking these delays into account.

[0138] It is not necessary to provide both mirrors 14 and 15, and either one may be provided. Either one of the mirrors may also serve to control the optical path of the light irradiated by irradiation device 11 and the optical path of the image captured by image capture device 12.

[0139] Next, other embodiments of the above-mentioned processing system will be described. Note that each embodiment described below can be combined with the above-mentioned first to third modified examples.

[0140] 2. Second Embodiment In the second embodiment, the processing system 10 is used for three-dimensional measurement of an object.

[0141] 19 is a diagram showing the overall configuration of a processing system according to the second embodiment. In this embodiment, an irradiation device 11 irradiates an object 100 with irradiation light, and an imaging device 12 acquires multi-viewpoint images while the object 100 is held and moved. A processing unit 132 measures the three-dimensional shape of the object 100 based on the emitted light contained in the acquired multi-viewpoint images, for example, using a known method such as depth from motion. Alternatively, a two-dimensional position sensing device (PSD) may be disposed on the surface irradiated with the irradiation light to detect the irradiation position of the irradiation light, and the three-dimensional coordinates of the object 100 may be calculated based on the detected irradiation position and the known relative position of the imaging device 12.

[0142] Furthermore, the processing system 10 is not limited to measuring the three-dimensional shape of the entire object 100, and may also measure, for example, the surface roughness (texture) of the object 100. It is believed that the emitted light is also affected by the change in the irradiation efficiency of the irradiated light depending on the inclination of the surface of the object 100. Therefore, the processing system 10 may calculate the inclination of the surface of the object 100 by detecting the emitted light.

[0143] When using conventional physical markers, measuring the entire shape of an object requires attaching multiple markers to the entire object. Furthermore, for example, when the object has a complex shape, the markers themselves may interfere with measurement depending on the imaging angle. In this regard, the processing system 10 can continuously attach new emissive light markers in real time even when the object 100 moves in an unknown and random manner, thereby enabling the measurement of the entire three-dimensional shape of the object 100 in a simple manner without the markers affecting the measurement. Furthermore, even for objects whose shape changes, such as flexible objects, emissive light markers can be attached and their shapes can be measured. The results of the shape measurement of the object 100 may be used, for example, for archiving.

[0144] In order to efficiently perform three-dimensional measurement, the irradiated light may be a two-dimensional pattern. If the object 100 is a flexible body, the two-dimensional pattern of the emitted light may change during measurement, so the irradiated light pattern may be updated for each frame, or the two-dimensional pattern may be continuously irradiated intermittently.

[0145] Even if the angle of view of the imaging device 12 has an angle with respect to the emitted light emitted from the object 100, if the area of ​​the object 100 from which the emitted light is emitted is flat, the center of the emitted light can be detected by finding the center of gravity of the emitted light reflected, for example, on a circle or ellipse.

[0146] Although a plurality of mutually independent irradiation light beams may be irradiated, a single point irradiation light beam or a two-dimensional pattern irradiation light beam is preferable in order to reduce calculation costs and avoid the processing system becoming complicated.

[0147] As in the first modification, the irradiated light may be time-modulated to blink periodically. By capturing the same emitted light with multiple image capture devices and observing the period, it is possible to synchronize the multiple image capture devices.

[0148] When emitting multiple independent beams of radiation, the radiation may be time-modulated to blink at different intervals to distinguish between them. In this case, the more types of emitted light to be distinguished, the lower the speed. Therefore, to ensure high speed, the threshold for detecting emitted light may be set relatively high and the emission duration of the emitted light may be intentionally shortened to shorten the radiation period. As the number of imaging frames increases, the number of distinguishable blinking periods increases, improving the accuracy of identification. Using the blinking interval of the radiation enables error detection, making it possible to distinguish between emitted light and light other than emitted light, thereby also improving the accuracy of emitted light detection. Note that if pulse code modulation is applied instead of time modulation, the speed may be further degraded, but the accuracy of emitted light detection is improved. These are also true for other embodiments described below.

[0149] 3. Third Embodiment In the third embodiment, the processing system 10 is used to visualize the trajectory of the movement of an object. When the object is, for example, a fluid including liquid and gas, or powder, it is difficult to attach a conventional physical marker without affecting the object itself, or to attach a marker accurately to the area where the object exists, making it difficult to track the movement of the fluid or powder. In this regard, the processing system 10 irradiates the fluid or powder with irradiation light, causing emitted light to be emitted from the fluid or powder itself. This makes it possible to attach an emitted light marker accurately to the object to be measured without affecting the fluid or powder, and visualize the trajectory of their movement.

[0150] Fig. 20 is a diagram showing the overall configuration of a processing system according to the third embodiment. In the operating environment shown in Fig. 20, powdered granulated sugar 100A, which is an example of a target object, is placed in a petri dish 30, and an obstacle 31 is fixed to the space by a fixing device 32. The petri dish 30 is placed on a rotating table (not shown), and when rotated in the direction of the arrow shown in Fig. 20, the obstacle 31 stirs the granulated sugar 100A in the petri dish 30.

[0151] The irradiation device 11 irradiates the granulated sugar 100A with irradiation light. The imaging device 12 captures the light emitted from the granulated sugar 100A. The mirror 16 is composed of a single-axis galvanometer mirror, and serves to control the optical path of the irradiation light by the irradiation device 11 and the optical path of the image capture by the imaging device 12. Images acquired in this operating environment are shown in Figures 21A to 22B. The irradiation time of the irradiation light by the irradiation device 11 is 10 milliseconds. The other conditions are the same as those in the experiment described using Figures 8A and 8B, so detailed explanations will be omitted.

[0152] Figures 21A and 21B are diagrams showing images captured by an imaging device when radiated light is irradiated onto granulated sugar outside an obstacle in the operating environment of the processing system shown in Figure 20. Specifically, Figure 21A is an image captured immediately after irradiation of the radiated light was stopped, and Figure 21B is an image captured 200 milliseconds after the image shown in Figure 21A was captured. Note that radiated light can also pass through granulated sugar, and as the radiated light is diffused, the light appears to have a certain degree of spread in Figures 21A and 21B. The same is true for Figures 22A and 22B, which will be described later.

[0153] From the transition of the emitted light 210 of the granulated sugar 100A shown in FIGS. 21A and 21B, it can be seen that the granulated sugar 100A spreads outward from the obstacle 31, tracing a trajectory.

[0154] Figures 22A and 22B are diagrams showing images captured by an imaging device when irradiation light is irradiated onto granulated sugar inside an obstacle in the operating environment of the processing system shown in Figure 20. Specifically, Figure 22A is an image captured immediately after irradiation of the irradiation light was stopped, and Figure 22B is an image captured 200 milliseconds after the image shown in Figure 22A was captured.

[0155] 22A and 22B, it can be seen that the granulated sugar 100A is pushed toward the inside of the Petri dish 30 by the obstacle 31 and is then diffused inward. In other words, the processing system visualizes the trajectory of the movement of the granulated sugar.

[0156] In addition, the flow of gel-like objects on a factory production line can be visualized, the flow can be optimized based on the visualized data, and the viscous resistance of the object can be calculated. For example, in the case of chocolate, it is difficult to visualize the chocolate flow even with thermography if there is no color difference between the solid and liquid and the temperature difference is small. The processing system 10 visualizes the flow regardless of the color or temperature of the chocolate, making it possible to detect areas with poor flow.

[0157] The illumination light may be a two-dimensional pattern to visualize flows at multiple positions simultaneously, or multiple independent illumination lights may be used to reduce calculation costs. Alternatively, a single illumination light may be distributed to multiple illumination positions using a galvanometer mirror.

[0158] The intensity distribution of emitted light may also be used to visualize the movement of a fluid. For example, if the attenuation characteristics of emitted light from an object are known, the velocity and acceleration of the object can be calculated by comparing the captured image with the attenuation characteristics. The velocity of the object can be calculated from at least one image capturing the trajectory of emitted light, and the acceleration of the object can be calculated from multiple images.

[0159] 4. Fourth Embodiment In the fourth embodiment, the processing system 10 is used to correct motion blur. Note that the overall configuration of this embodiment can be inferred from, for example, Fig. 19, and therefore is not shown.

[0160] When capturing an image, motion blur due to camera shake or object movement can occur, degrading image quality. To address this degradation, a known method is deconvolution, which uses a blur kernel as an auxiliary to perform deconvolution after capturing an image. To obtain this blur kernel, a known method is to incorporate an acceleration sensor into the image capture device and detect the trajectory of the image capture device's movement. However, this method cannot reflect the movement of the object. Furthermore, the need for double integration can result in a decrease in accuracy.

[0161] In this regard, by using the processing system 10, by attaching an emission light marker to the object 100, it is possible to acquire the trajectory of the object's movement along with the image of the object 100, and a relative blur kernel is calculated from the trajectory of the object's movement. That is, if the trajectory of the emission light is a point spread function (PSF), and the PSF is convolved with an image without motion blur to form a blurred image, the blur can be removed by, for example, a deconvolution operation. Note that the algorithm for removing blur is not limited to a simple deconvolution operation, and other algorithms (e.g., the Lucy-Richardson algorithm) that improve accuracy by knowing the PSF may also be combined.

[0162] According to this embodiment, since the blur kernel can be obtained from the trajectory of the emitted light, the calculation cost can be reduced and highly accurate results can be obtained compared to when performing blind deconvolution without using a blur kernel. Furthermore, in this embodiment, since the speed of the object can be calculated from the brightness of the emitted light, the object may accelerate or decelerate during the exposure time of the imaging device.

[0163] Fig. 23A is a diagram showing an image captured when the sketchbook is irradiated with irradiation light. Fig. 23B is a diagram showing an image captured when the sketchbook is moving. Fig. 23C is a diagram showing the trajectory of emitted light from the sketchbook. Fig. 23D is a diagram showing an image reconstructed by deconvolution.

[0164] From FIG. 23B, it can be seen that the movement of the sketchbook causes blurring, resulting in a blurred edge of the sketchbook (see the circled area in FIG. 23B). FIG. 23C shows the trajectory of the emitted light captured by an imaging device for capturing the emitted light (corresponding to imaging device 12 of processing system 10), separate from the imaging device for capturing the sketchbook. FIG. 23D is a diagram reconstructed by performing a deconvolution operation on the image of FIG. 23B using the trajectory. From the image of FIG. 23D, it can be seen that the edge of the sketchbook is clearer than in the image of FIG. 23B (see the circled area in FIG. 23D). Note that the higher the spatial resolution of the image capturing the emitted light, the more accurate the blur kernel, and therefore the more accurately the blur is removed.

[0165] 5. Fifth Embodiment In the fifth embodiment, the processing system 10 is used for motion capture. Possible targets for motion capture include, but are not limited to, humans and organisms, including randomly walking microorganisms. For example, when estimating a person's posture using motion capture, the person may be irradiated with light and the emitted light may be used as a marker. When multiple independent emitted light markers are attached, the above-mentioned modulation method may be used to distinguish between the multiple emitted lights. Note that, since the processing after detecting the emitted light can use the same method as existing motion capture, a description using figures will be omitted.

[0166] The processing system 10 can apply emitted light markers even in tiny areas that are unsuitable for physical markers or areas where the distance between markers cannot be sufficiently secured, thereby achieving high scalability.

[0167] Markers used in conventional motion capture generally have three-dimensional shapes, which can restrict movement, such as rubbing between markers while a person is moving. In this regard, the processing system eliminates these restrictions. The emitted light used as the markers stays on the person and follows their movements without delay, allowing for more dynamic movements. Furthermore, the processing system 10 potentially enables posture estimation with higher accuracy than physical markers, since the emitted light contains time information.

[0168] Furthermore, while physical markers can spoil the appearance, the processing system 10 makes it possible to select the position and timing for attaching emissive light markers. Therefore, for example, in a situation where a person to be estimated and an audience member are in the same space, pose estimation can be performed without spoiling the appearance by attaching emissive light markers at positions and times that are not noticeable to the audience. For example, the positional relationship between the target person and the audience member may be measured in real time, and emissive light markers may be attached at positions that are not visible to the audience member.

[0169] 6. Sixth Embodiment In the sixth embodiment, the processing system 10 is used for SLAM (Simultaneous Localization and Mapping). Note that the overall configuration of this embodiment can be inferred from, for example, FIG.

[0170] In this embodiment, for example, while the processing system 10 moves relative to the surrounding environment, the irradiation device 11 irradiates an object in the surrounding environment with irradiation light. The imaging device 12 uses the emitted light generated by the irradiation as a marker, and the processing unit 132 can create an environmental map or estimate its own position based on the obtained information.

[0171] Conventional SLAM uses landmarks or QR codes (registered trademark) placed in advance in the surrounding environment as markers, so errors can occur in the calculated three-dimensional position depending on the angle at which the image is captured, for example, when the landmarks or QR codes have a certain size. In this regard, by using the processing system 10, for example, by irradiating a point-like illumination light, the emitted light can be accurately detected even if the viewpoint of the image capture is different, improving measurement accuracy.

[0172] The processing system 10 can use emitted light as a marker, thereby expanding the area that can be marked. Since it can distinguish between moving and stationary objects in the surrounding environment, it can, for example, assign emitted light markers to objects that move regularly when creating an environmental map.

[0173] 7. Seventh Embodiment In the seventh embodiment, the processing system 10 is used to measure the mechanical properties of an object.

[0174] FIG. 24 is a diagram showing the overall configuration of a processing system according to a seventh embodiment. In this embodiment, the processing system 10 attaches an emission light marker to a predetermined position on the object 100. The object 100 with the attached emission light marker is mounted on a vibration table 33 that vibrates in the direction of the arrow shown in FIG. 24 and vibrates together with the vibration table 33. By detecting the movement of the emitted light during this process using the imaging device 12, it is possible to measure the local responsiveness of the object to vibration at a predetermined position (e.g., stress distribution in response to an external field, high-speed transient phenomena when stress is applied, etc.). By repeating this process at various positions on the object, it is possible to acquire the mechanical properties of the object 100, estimate the relative orientation of the object 100 with respect to the imaging device 12, and create a feature map of the object 100.

[0175] In the past, when analyzing the mechanical properties of an object, it was necessary to attach physical markers or strain sensors to the object, but attaching such objects to the object could affect the mechanical properties of the object. In this regard, the processing system 10 can attach emitted light markers without affecting the mechanical properties of the object, thereby improving the measurement accuracy of mechanical properties, etc. In addition to responsiveness due to vibration, various mechanical properties can be obtained by measurements involving translational and rotational motion.

[0176] 8. Eighth Embodiment In the eighth embodiment, the processing system 10 is used to determine whether an object has crossed a boundary line.

[0177] FIG. 25 is a diagram showing the overall configuration of a processing system according to the eighth embodiment. In FIG. 25, the control device 13 of the processing system 10 is not shown. The irradiation device 11 can create a boundary line or boundary surface using irradiation light that separates at least two regions by irradiating irradiation light linearly or planarly along a certain straight line or plane. As shown in FIG. 25, in this embodiment, a boundary line using irradiation light is set at the entrance and exit of the box 40. When a part 100B, which is an example of an object, exits the box 40 and touches the boundary line, emitted light is emitted from the point of contact. In other words, the part 100B is automatically marked.

[0178] By detecting the emitted light with the imaging device 12, the processing unit 132 can determine whether the part 100B has touched the boundary line and determine which part of the part 100B has crossed the boundary line. Furthermore, even after the part 100B has left the box 40, it is possible to obtain information that the part 100B was in the box 40, making it possible to distinguish between parts that are similar in appearance but different in an assembly plant, for example. It is also possible to mark the parts that have entered the box 40 from outside the box 40.

[0179] For example, it can be used to detect objects that have protruded from their proper area in a factory lane, or to identify parts of an object that have protruded into a contaminated area, so that the relevant parts can be cut off and the remaining parts can be reused. Alternatively, for example, by marking a finger or pen that enters the boundary of the irradiated light, the boundary can be given the function of a touch display. By attaching an emitted light marker to a finger or pen, it is possible to determine the movement of the finger that touched, the length of time that the touch was made, or how many fingers were used to touch.

[0180] While it is difficult to automatically attach a marker to a predetermined object using conventional physical markers, the processing system 10 can automatically attach an emitted light marker to a predetermined object using a relatively simple configuration that involves irradiating the object with irradiation light.

[0181] It is possible to provide information about where an object comes from by forming multiple boundaries or boundary surfaces with different wavelengths, intensities, periods, etc. of the irradiated light. The wavelength, intensity, period, etc. of the irradiated light may vary continuously on the spatial axis.

[0182] 9. Ninth Embodiment In the ninth embodiment, a processing system 10 is used to present a target point. Note that the overall configuration of this embodiment can be inferred from, for example, FIG.

[0183] For example, in situations where the target position (e.g., the part connection position) moves unstably, such as during sealing work in which a robot connects parts together, it is difficult to accurately recognize the target position. By using the processing system 10, the target point can be assigned quickly by using emitted light as the target point. Even if the object moves randomly or deforms after the target point is assigned, the target point can track without deviation and present the target position. While physical markers can get in the way during sealing work, the processing system 10 eliminates this risk and can assign target points to parts of various shapes without leaving any trace of the target point.

[0184] In the above-described embodiment and modified examples, the object 100 emits emitted light such as fluorescence or phosphorescence (see "Photoluminescence" in FIG. 26), but the present invention is not limited to this. For example, the object 100 may be irradiated with light (light energy, photoexcitation), absorb light, and convert it into thermal energy (wavelength conversion), thereby emitting emitted light (infrared rays) (see "Thermoluminescence" in FIG. 26). According to this aspect, emitted light (infrared rays) is emitted due to the temperature rise phenomenon of the object 100, and therefore, similar to the above-described embodiment and modified example, an emitted light marker can be attached robustly to the appearance and shape of the object 100. Below, an aspect in which infrared rays are emitted as emitted light will be described in detail.

[0185] 10. Tenth Embodiment In the tenth embodiment, the surface of an object (assumed to be black paper here) 100 is instantly heated using laser heating, and an emitting light marker that does not need to be attached is generated by capturing an image of the surface of the object 100 using a thermal camera.

[0186] Fig. 27 is a diagram showing the configuration of the main parts of a processing system 10 according to the tenth embodiment. Note that in Fig. 27, the control device 13 of the processing system 10 is not shown. Irradiation device 11 is, for example, a semiconductor visible light laser that irradiates laser light with a wavelength of 640 nm. Mirror 17 is, for example, a two-axis galvanometer mirror, and its operation is controlled by control device 13. Control device 13 controls mirror 17 to generate emitted light markers of any shape or pattern on the surface of object 100. Imaging device 12 is, for example, a thermal camera with high temperature resolution and high spatial resolution, and captures images of the emitted light markers generated on the surface of object 100 (in other words, emitted light emitted from the surface of object 100).

[0187] Figure 28 is a diagram illustrating the temperature change when a laser is irradiated onto the object 100 for different irradiation times in the operating environment of the processing system 10 shown in Figure 27, with the vertical axis representing temperature and the horizontal axis representing elapsed time. As shown in Figure 28, the longer the irradiation time, the longer the heat dissipation time, and the longer the time it can be detected as an emitted light marker. Note that the longer the irradiation time, the higher the temperature; for example, with an irradiation time of 50 ms, the temperature exceeded the left axis in the figure, but this is shown as an omitted range. In this experiment, a threshold value Tth at which it can be detected as an emitted light marker was set based on the temperature changes due to heating and heat dissipation, and an emitted light marker was generated by irradiating a laser at a specified location on the object 100. The laser was then irradiated at another location until the temperature at the specified location reached (or decreased to) the marker detection threshold value Tth. Then, when the temperature at the specified location reached the marker detection threshold value Tth, the laser returned to the specified location and irradiated again, making it possible to maintain the temperature required to detect the emitted light marker.

[0188] [Table 1] Table 1 shows the relationship between the laser irradiation time ti and the heat dissipation time td in this experiment. As shown in Table 1, when the irradiation time ti is set to 10 ms, it is possible to heat up to 33 locations during heat dissipation as an integer value of td / ti. In other words, when the irradiation time ti is set to 10 ms, it is possible to generate up to 33 emitted light markers almost simultaneously.

[0189] Fig. 29 shows experimental results when the irradiation time ti was set to 10 ms and various patterns (drawing patterns) of emitting light markers were generated on the object 100. Specifically, Fig. 29A shows experimental results when a point cloud (dot row) was drawn as the emitting light marker, and Fig. 29B shows experimental results when the kanji character "Hikari" (light) was drawn as the emitting light marker. Each image shown in Fig. 29 is generated by the imaging device 12 or the like.

[0190] As shown in Figure 29A, when a dot row is drawn, the dots in the horizontal direction can be distinguished, but the dots in the vertical direction, which are closer together than the horizontal dots, cannot be distinguished. This is presumably because the temperature propagates to the surrounding area after heating.

[0191] On the other hand, as shown in Figure 29B, when the kanji character "Hikari" was drawn, the drawn character (i.e., "Hikari") was clearly readable in both the thermal image and the binary image. As can be seen in the binary image, the bottom left dots that make up the drawn character (i.e., the dots drawn earlier in the drawing order) are faded, so under the conditions of this experiment, it is considered desirable to set the drawing range to approximately 30 dots (= 6 dots horizontally × 5 dots vertically) or less. Furthermore, in this experiment, the complex shape of kanji was drawn by combining dots, but this method can also be applied to the continuous drawing of lines, circles, etc.

[0192] Furthermore, the object 100 is not limited in any way as long as it absorbs the irradiated light and converts the energy into, for example, thermal energy. Specifically, it is not limited to the black paper exemplified in this embodiment (tenth embodiment), but it goes without saying that it can also be applied to, for example, an acrylic plate, a black absorbing material, or paper of various colors (chromatic or achromatic). Furthermore, the wavelength of the semiconductor visible light laser, the laser irradiation time, etc. can be arbitrarily set and changed depending on the desired drawing pattern, the type of object 100, etc.

[0193] <Other application examples> In addition to the above-described embodiments, the processing system 10 may be used for a variety of other purposes.

[0194] The processing system 10 may be applied, for example, to an inspection system that inspects an external environment while moving at high speed relative to the external environment. Sensing of the external environment from a moving object becomes more efficient the faster the moving object moves. However, when a moving object moves at high speed, motion blur occurs, making it necessary to compensate for the relative movement of the object with high accuracy. In this regard, the present processing system can calculate the angular velocity of the object, thereby enabling highly accurate and stable sensing of the external environment. Specifically, the system can be used, for example, for inspecting infrastructure such as roads and trains, or for inspecting moving lines such as factory lines. In this case, inspection can be performed with higher efficiency than conventional inspection systems.

[0195] The processing system 10 may be used, for example, for object detection under poor lighting conditions. For example, even in an environment where a strong spotlight causes overexposure and makes it impossible to capture images using a conventional imaging device, the object can be detected by irradiating the object with ultraviolet light as the irradiating light and capturing an image using a filter that transmits the wavelength of the emitted light. By temporally modulating the irradiating light and thinning out a predetermined number of frames, a specific object can be selectively visualized. As described in the tenth embodiment above, the surface of the object may be instantaneously heated using laser heating and then imaged with a thermal camera, thereby selectively visualizing the object.

[0196] In the processing system 10, by capturing images using multiple imaging devices through filters with different wavelength characteristics, it is possible to selectively acquire and present information in each imaging device, which can be used for entertainment and information communication, for example.

[0197] The processing system 10 may be used to track multiple objects, for example. For example, it may track pedestrians on a road where hundreds of people are walking. By emitting light so that the entire floor is illuminated and attaching emitted light markers to each person's feet, the emitted light can be tracked continuously while the person is walking.

[0198] The above-described embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The elements of the embodiments, as well as their arrangement, materials, conditions, shapes, sizes, etc., are not limited to those illustrated and can be modified as appropriate. Furthermore, configurations shown in different embodiments can be partially substituted or combined with each other. [Explanation of symbols]

[0199] 10, 10A to 10C... processing system, 11... irradiation device, 12... imaging device, 13, 13A, 13B... control device, 14 to 16... mirror, 20... CPU, 21... RAM, 22... ROM, 23... communication unit, 24... input unit, 25... display unit, 30... petri dish, 31... obstacle, 32... fixture, 33... vibration table, 40... box, 100... object, 100A... granulated sugar, 100B... part, 110... irradiation unit, 120... imaging unit, 130... setting unit, 131... detection unit, 132... processing unit, 133... modulation unit, 134... matching unit, 200, 210, 220... emitted light

Claims

1. an irradiation unit that irradiates an object with irradiation light; an imaging unit that images emitted light emitted from the object itself in response to irradiation with the irradiation light; a detection unit that detects the emitted light as a marker of the object based on the image captured by the imaging unit; a processing unit that executes a predetermined process based on the emitted light detected by the detection unit; Equipped with The processing unit further includes a matching unit that matches markers in each image detected by the detection unit, the irradiation light includes pattern light spreading on a two-dimensional plane, the imaging unit captures images at a plurality of timings using emitted light emitted from the object itself in response to the irradiated light as a marker; the matching unit performs marker matching based on a plurality of images captured over a plurality of timings; The imaging unit includes a high-speed camera that captures the emitted light at a frame rate of 100 fps or more.

2. the processing unit measures at least one of a relative position, a relative orientation, and a relative velocity of the object with respect to the imaging unit, and tracks the object. The processing system of claim 1 .

3. the imaging unit images emitted light emitted from the object itself in response to a certain irradiation at a plurality of timings, the detection unit detects the emitted light from each of a plurality of images captured over the plurality of times; the processing unit calculates a relative velocity of the object with respect to the imaging unit based on a transition of the position of the emitted light between the plurality of images. The processing system of claim 2 .

4. the imaging unit captures an image including a plurality of emitted lights respectively emitted from the object itself in response to irradiation at a plurality of timings, the detection unit detects the plurality of emitted lights from the image; the processing unit calculates a relative velocity of the object with respect to the imaging unit based on intervals between the plurality of emitted lights. The processing system of claim 2 .

5. The irradiation unit continuously irradiates the irradiation light, The imaging unit captures an image including emitted light emitted from the object itself in response to continuous irradiation, the detection unit detects a trajectory of the emitted light, the processing unit calculates a relative velocity of the object with respect to the imaging unit based on the trajectory of the detected emitted light. The processing system of claim 2 .

6. the imaging unit acquires a multi-viewpoint image including the emitted light, The processing unit measures the shape of the object based on emitted light included in the acquired multi-viewpoint image. The processing system of claim 1 .

7. the processing unit visualizes a trajectory of movement of the object using the detected emitted light. The processing system of claim 1 .

8. the processing unit calculates a trajectory of movement of the object based on the detected emitted light, and corrects motion blur in an image including the object based on the calculated trajectory. The processing system of claim 1 .

9. The processing unit performs motion capture based on the detected emitted light. The processing system of claim 1 .

10. the processing unit performs simultaneous localization and mapping (SLAM) based on emitted light detected from the object in the surrounding environment of the processing system. The processing system of claim 1 .

11. The processing unit measures mechanical properties of the object based on the trajectory of the detected emitted light. The processing system of claim 1 .

12. the irradiation unit irradiates the irradiation light along a boundary surface separating at least two regions, the processing unit determines whether the object has touched the boundary surface based on the emitted light emitted from the object when the object comes into contact with the irradiated light. The processing system of claim 1 .

13. Further provided is a light path control unit that guides the irradiation light irradiated from the irradiation unit to an arbitrary region of the object. A processing system according to any one of claims 1 to 12.

14. The imaging unit includes a high-speed camera that captures the emitted light at a frame rate of 100 fps or more. A processing system according to any one of claims 1 to 13.

15. the imaging unit images emitted light emitted from the object itself after stopping the irradiation of the irradiation light on the object; A processing system according to any one of claims 1 to 14.

16. The emitted light includes delayed fluorescence, phosphorescence, afterglow, phosphorescence, or infrared light emitted from the object itself after irradiation with the irradiation light.

16. A processing system according to any one of claims 1 to 15.

17. an illumination step of irradiating an object with illumination light; an imaging step of imaging emitted light emitted from the object itself in response to irradiation with the illumination light; a detecting step of detecting the emitted light as a marker of the object based on the captured image; an execution step of executing a predetermined process based on the detected emitted light; The executing step further includes a matching step of matching markers in each image detected by the detecting unit; the irradiation light includes pattern light spreading on a two-dimensional plane, the imaging step is performing imaging at a plurality of timings using emitted light emitted from the object itself in response to the irradiated light as a marker; The matching step performs marker matching based on a plurality of images captured over a plurality of timings; The imaging step includes imaging the emitted light at a frame rate of 100 fps or more using a high-speed camera.

18. Computer, an irradiation unit that irradiates the object with irradiation light; an imaging unit that images emitted light emitted from the object itself in response to irradiation with the irradiation light; a detection unit that detects the emitted light as a marker of the object based on the image captured by the imaging unit; and a processing unit that executes a predetermined process based on the emitted light detected by the detection unit; A processing program that functions as The processing unit further includes a matching unit that matches markers in each image detected by the detection unit, the irradiation light includes pattern light spreading on a two-dimensional plane, the imaging unit captures images at a plurality of timings using emitted light emitted from the object itself in response to the irradiated light as a marker; the matching unit performs marker matching based on a plurality of images captured over a plurality of timings; The imaging unit includes a high-speed camera that captures the emitted light at a frame rate of 100 fps or more.

Images