Object detection system with time-of-flight sensor

Abstract

Object detection system (1), comprising: a time-of-flight sensor (10) which outputs an image of a target space based on a phase difference between reference light emitted toward the target space and light reflected from the target space, an object detection device (21) which detects a position of an object (W) present in the target space on the basis of the output image, an imaging condition calculation device (22) which calculates imaging conditions based on an image of the detected object (W), including an integration time and / or a light emission period of the time-of-flight sensor (10), an imaging condition changing device (23) which changes a configuration of the time-of-flight sensor (10) according to the calculated imaging conditions, a position / posture image calculation device (24) which calculates at least one image position based on the image of the detected object (W), a movement mechanism (30) which changes a position of the time-of-flight sensor (10) and / or the object (W) to the calculated imaging position, wherein the object detection device (21) further calculates an evaluation value indicating the degree of similarity of the object (W) on the basis of the output image and pre-stored reference data relating to the object (W) or a characteristic part of the object (W), wherein, if the calculated evaluation value is smaller than a predetermined threshold value, the imaging condition calculation device (22) calculates the imaging conditions or the imaging position / posture calculation device (24) calculates the imaging position, and wherein the object detection device (21) detects a position of the object based on the image output according to the changed imaging position and / or the changed imaging conditions.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention

[0001] The present invention relates to an object detection system and, more particularly, to an object detection system using a time-of-flight sensor. 2. State of the art

[0002] Time-of-flight (TOF) sensors are known as distance measuring devices that measure the distance to an object. These sensors output distances based on the time of flight of radiation. Time-of-flight sensors often use a phase difference method (a so-called "indirect method") in which a target area is irradiated with reference radiation, the intensity of which is modulated over predetermined cycles. The distance to the target area is calculated based on the phase difference between the emitted reference radiation and the radiation reflected from the target area. This phase difference is obtained from the amount of reflected radiation received. The following publications are known as the state of the art regarding time-of-flight sensors.

[0003] The published Japanese patent application (Kokai) JP 2012-123 781 A describes that in order to estimate the position and posture of a target object with high accuracy while reducing the influence of noise in a distance image due to multiple reflections or the like, image features are detected from a two-dimensional image corresponding to the distance image and the coordinates of the image features are calculated from the distance image.

[0004] The published Japanese patent application (Kokai) JP 2013-101 045 A describes that in order to recognize a three-dimensional position and posture of an object with low computational complexity, an area for searching the object in a two-dimensional image is set with a camera as a search condition on the basis of three-dimensional information obtained with an area sensor, two-dimensional position information of the object is obtained from the two-dimensional image within the set search conditions and three-dimensional point data is used to recognize the three-dimensional position and posture of the object, which are selected from the three-dimensional information on the basis of the obtained two-dimensional position information.

[0005] The published Japanese patent application (Kokai) JP 2011-64 498 A describes that in the time-of-flight method, less light reaches distant objects when the light is also emitted to objects at a short distance, which reduces the accuracy of the distance measurement with regard to the distant objects, whereas when objects at a long distance are exposed, the amount of light that reaches objects at a short distance increases, which causes saturation of the pixel charge, which is why imaging conditions are set in advance based on distance information and the objects are photographed according to the preset imaging conditions.

[0006] The published Japanese patent application (Kokai) JP 10-197 635 A describes that in order to improve the sensitivity and to obtain stable distance measurements even in difficult environments, such as rain, snow and glass contamination, environmental conditions are detected and the light emission time for the emitted light is changed according to the detected environmental condition to change the detection time for the light.

[0007] Published American patent US 9,739,601 B1 describes methods for determining a distance associated with a surface using the time of flight (ToF) of light pulses occurring at a predetermined frequency. A ToF camera may include a light transmitter that emits light according to one or more phase delays related to the configuration of a first of a plurality of memories of the ToF camera to receive light, convert it into electricity, and store a charge corresponding to the electricity. For example, a light transmitter may emit pulses with a phase delay of 0° (i.e., simultaneous with the opening of the first storage device), a phase delay of 90°, a phase delay of 180°, and a phase delay of 270°.The light captured in the storage devices and stored as energy can then be analyzed to estimate the distance to the surface of an object in an environment. After estimating the distance between the ToF camera and a surface, the ToF camera can optimize its phase delay to reduce the error associated with subsequent distance measurements.

[0008] Published German patent application DE 10 2016 107 959 A1 describes implementations of devices and techniques that provide multipath interference cancellation for imaging devices and systems. Various implementations use structured light to reduce, if not cancel, interference. For example, the light can be structured with respect to amplitude or phase based on the light's emission angle.

[0009] American patent application US 2013 / 0050426 A1 describes a method for extending the dynamic range of a depth map by deriving depth information from a synthesized image of a plurality of images acquired at different light intensity levels and / or over different sensor integration times. In some embodiments, a first image of an environment is acquired while the environment is illuminated with light of a first light intensity. Then, one or more subsequent images are acquired while the environment is illuminated with light of one or more different light intensities. The one or more different light intensities can be dynamically configured based on a degree of pixel saturation associated with previously acquired images.The first image and the one or more subsequent images may be synthesized into a synthesized image by applying high dynamic range imaging techniques.

[0010] Whenever “light” is mentioned in this description and in the claims, this always generally includes “electromagnetic radiation”. BRIEF DESCRIPTION OF THE INVENTION

[0011] Time-of-flight sensors have several disadvantages that arise from the very nature of their distance measurements. One disadvantage is the distance and reflectivity of the image object. Generally, fluctuations occur due to the influence of so-called shot noise during distance measurement with the time-of-flight sensor. Although the fluctuation range can be reduced by amplifying the radiation emission or the like for the reference radiation, distance measurement may fail if the reference radiation is too strong for objects at close range or objects with strong reflectivity, in which case so-called saturation or halation may occur. For objects at farther distances or objects with low reflectivity, however, the amount of reflected light from the reference light may be insufficient, thereby reducing the accuracy of the distance measurement.

[0012] Due to the distance measurement method of time-of-flight sensors, which detect the phase differences between reference radiation and reflected radiation, the fluctuation range is reduced as the light emission period of the reference radiation is shortened (the light emission frequency is higher), and when the radiation emission period of the reference radiation is reduced (the light emission frequency is increased), the distance measurement range is reduced, resulting in distance measurement errors due to the so-called "aliasing".

[0013] Another disadvantage of time-of-flight sensors is the effect of reflections on multiple paths (multiple reflections). For example, if a highly reflective object is located nearby, the latter object will be irradiated with reference radiation, which is reflected by the highly reflective object, shifting the distance measurement value of the object backward.

[0014] Therefore, there is a need for a technique to accurately detect the position of an object using a time-of-flight sensor.

[0015] A first embodiment of the present description provides an object detection system comprising a time-of-flight sensor which outputs an image of a target space based on a phase difference between reference radiation emitted to the target space and radiation reflected from the target space, further comprising an object detection device which detects a position of an object present in the target space based on the output image, an imaging condition calculation device which calculates imaging conditions including at least an integration time and / or a light emission period of the time-of-flight sensor based on an image of the detected object, an imaging condition changing device which changes a configuration of the time-of-flight sensor to the calculated imaging conditions, an imaging calculation unit for the position / pose which calculates at least an imaging position of the time-of-flight sensor based on an image of the detected object,and a movement mechanism that changes at least the position of the time-of-flight sensor and / or the object according to the calculated imaging position, wherein the object detection device further calculates an evaluation value indicating the degree of similarity of the object based on the output image and pre-stored reference data regarding the object or a characteristic part of the object, wherein, if the calculated evaluation value is less than a predetermined threshold value, the imaging condition calculation device calculates the imaging conditions or the position / posture imaging calculation device calculates the imaging position, and wherein the object detection device detects a position of the object based on the image output according to the changed imaging position and / or the changed imaging conditions.

[0016] A second embodiment of the present description provides an object detection system comprising a time-of-flight sensor which outputs an image of a target space based on a phase difference between reference radiation emitted to the target space and radiation reflected from the target space, further comprising an object detection device which detects a position of an object present in the target space based on the output image, an imaging condition calculation device which calculates imaging conditions including at least an integration time and / or a light emission period of the time-of-flight sensor based on an image of the detected object, an imaging condition changing device which changes a configuration of the time-of-flight sensor to the calculated imaging conditions, an imaging position / posture calculation unit which calculates at least one imaging position of the time-of-flight sensor based on an image of the detected object,and a movement mechanism which changes at least the position of the time-of-flight sensor and / or the object according to the calculated imaging position, wherein the imaging position / posture calculation means calculates the imaging position such that the detected object or a characteristic part of the object is magnified in the image, and wherein the object detection means detects a position of the object based on the image output according to the changed imaging position and / or the changed imaging conditions. BRIEF DESCRIPTION OF THE CHARACTERS Fig. 1 is a block diagram showing the structure of an object detection system according to an embodiment. Fig. Figure 2 shows the time sequences for radiation reception and radiation emission for a typical time-of-flight sensor. Fig. 3 shows an example of a calculation method for mapping conditions (integration time). Fig. 4 shows a modified example of a calculation method for mapping conditions (integration time). Fig. 5 shows a modified example of a calculation method for mapping conditions (integration time). Fig. Figure 6 shows another modified example of a calculation method for mapping conditions (integration time). Fig. Figure 7 shows another modified example of a calculation method for mapping conditions (integration time). Fig. Figure 8 shows an example of the occurrence of a distance measurement error due to aliasing and a method for calculating an appropriate radiation emission frequency. Fig. 9 shows an example of a method for calculating an image position. Fig. Figure 10 shows an example of images in which an object is captured before and after a change in position with magnification. Fig. 11A shows a perspective view of an example of a preset imaging position and an imaging posture with respect to an object. Fig. 11B shows a perspective view of an example of a preset imaging position and an imaging posture with respect to an object. Fig. 12A shows an imaging position in which the surface of an object can be seen, which has the effect of a multi-path. Fig. Figure 12B shows an imaging position in which the surface of an object can be seen, which has the effect of a multipath. Fig. Figure 13A shows an example of a movement mechanism. Fig. Figure 13B shows a modified example of a movement mechanism. Fig. 14 is a flowchart illustrating the operation of an object detection system according to an embodiment. Fig. 15 is a block diagram of a modified example of the structure of an object detection system. DESCRIPTION OF DETAILS

[0017] Embodiments of the present invention will now be described in more detail with reference to the accompanying figures. In the figures, identical or similar components are designated by the same reference numerals. The embodiments described below do not limit the technical scope of the invention, nor do they limit the meaning of the terms in the claims.

[0018] Fig. 1 shows the structure of an object detection system 1 according to the present embodiment. The object detection system 1 is, for example, a robot system that uses a time-of-flight sensor 10 for the robot's vision. The system may therefore comprise a time-of-flight sensor 10, a controller 20 connected to the time-of-flight sensor 10 via a wire or wirelessly, and a robot 10 and a tool 31 controlled by the controller 20.

[0019] The time-of-flight sensor 10 can be a time-of-flight camera with a plurality of light-emitting elements and a plurality of light-receiving elements, or it can also be a laser scanner with a light-emitting element and a light-receiving element, or the like. The time-of-flight sensor 10 uses the so-called "indirect method" and outputs a distance image of the target space based on the phase difference between the reference radiation S emitted toward the target space and the radiation reflected from the target space. The time-of-flight sensor 10 can output a radiation intensity image of the radiation received from the target space. The distance image and the radiation intensity image can be grayscale images, depending on the visibility conditions, or RGB (red, green, blue) images in which the magnitude of the numerical value is represented by the color density, or they can simply be an array of numerical values.For the sake of simplicity of illustration, all image data output by the time-of-flight sensor 10 are referred to below as “image”.

[0020] The controller 20 may be a conventional computing device including a CPU (central processing unit), an FPGA (field-programmable gate array), an ASIC (application-specific integrated circuit), or the like, or it may be a quantum computer including a quantum processor. The controller 20 is configured to detect the position (and, if necessary, also the posture) of an object W based on the image output from the time-of-flight sensor 10, and to correct the operation of the robot 30 as part of the movement mechanism and the operation of the tool 31 as part of the movement mechanism, which are learned in advance based on the detected position of the object W.

[0021] The robot 30 may be an articulated robot or it may be another industrial robot, such as a parallel-jointed robot. The tool 31 may be a hand with suction that sucks an object W, or it may be a hand with multiple fingers, or it may be another tool depending on the task to be performed on the object W. Examples of other tools include sealing tools, welding tools, screw tightening tools, soldering tools, laser processing tools, etc.

[0022] Fig. 2 shows the time sequence of radiation emission and reception of the exemplary time-of-flight sensor 10. The time-of-flight sensor 10 emits reference radiation toward the target space according to predetermined radiation emission periods T, receives the radiation reflected from the target space according to a plurality of sampling time sequences, and obtains the respective charge amounts Q1 to Q4. The sampling time sequences may have phases that are offset, for example, by 0°, 90°, 180°, and 270° with respect to the reference radiation, but the sampling time sequence is not limited to this. The time-of-flight sensor 10 calculates the distance d of the target space and the reflected intensity s of the reference radiation for each pixel based, for example, on the following known equation. In the equation, c is the speed of light and f is the radiation emission frequency of the reference radiation.Different methods for calculating the distance d and the reflection intensity s of the reference radiation are known and the present description is not limited to one calculation method.

[0023] For the sake of simplicity, the term “light” is used below to refer to radiation in general, which covers all suitable electromagnetic radiation. d=c2πf⋅arctan(Q2−Q4Q1−Q3) s=|Q1−Q3|+|Q2−Q4|

[0024] The time-of-flight sensor 10 performs imaging under predetermined imaging conditions and at a predetermined imaging position. The imaging conditions include the integration time IT, the light emission period T of the reference light, the light intensity of the reference light, the required imaging repetitions N, and the like. The integration time IT corresponds to the integration time in which a charge amount Qn is accumulated. Instead of the light emission period T of the reference light, the light emission frequency f (= 1 / T) of the reference light can also be used. The light intensity of the reference light can be adjusted by the number of light-emitting elements, or it can be adjusted by a voltage value or a current value applied to the light-emitting elements.The required imaging repetitions N correspond to the number of images generated by imaging with the time-of-flight sensor 10, which are required to combine a plurality of images (for example, for pixel averaging). This enables, for example, the time-of-flight sensor 10 to perform continuous high-speed recording operation that is independent of the normal frame rate. The N images can be combined on the time-of-flight sensor 10 side, or they can also be combined on the controller 20 side (or in a main processing unit 50, which is described in more detail below). In the first case, the image output by the time-of-flight sensor 10 is a composite image. Furthermore, a time filter constant N can be used instead of the required imaging repetitions N."Time filter" here refers to an image processing filter applied to a plurality of time-series images, and the term "time filter constant N" corresponds to the number of time-series images. The time filter can be implemented on the time-of-flight sensor 10 side or on the controller 20 side (or in a host computer 50, as described in more detail below).

[0025] Accordingly Fig. 1, the object detection system 1 includes an object detection device 21 which detects the position of the object W (including a posture, as required; this also always applies to the following) on ​​the basis of the image output from the time-of-flight sensor 10, an imaging condition calculation device 22 which calculates the imaging conditions on the basis of an image of the detected object W, and an imaging condition changing device 23 which adapts the configuration of the time-of-flight sensor 10 to the calculated imaging conditions, wherein the object detection device 21 detects the position of the object W on the basis of the image output under the adapted imaging conditions.

[0026] Furthermore, the object detection system 1 comprises an imaging position / pose calculation device 24 that calculates an imaging position (including an imaging pose, if necessary; this also applies to all subsequent determinations of "position") based on the image of the detected object W, and a movement mechanism that changes the position of the time-of-flight sensor 10 or the object W (including the pose, if necessary; this addition to "position" also applies to all subsequent related features) according to the calculated imaging position. In this case, the object detection device 21 detects the position of the object W based on the image output according to the changed imaging position and / or the changed imaging conditions.The movement mechanism may be a robot 30 that supports the time-of-flight sensor 10, or it may also be a robot 30 that supports the object W. The robot 30 changes the position of the time-of-flight sensor 10 or the object W according to the calculated imaging position.

[0027] According to the first embodiment, the object detection device 21 detects the position of the object W by calculating an evaluation value indicating the degree of similarity of the object W based on the image output from the time-of-flight sensor 10 and reference data of the object W or a characteristic part of the object W stored in advance in the storage device 25. The object detection device 21 can detect the position of the object W using at least one of the known object detection methods, for example, a matching method for matching pixels, blob analysis for analyzing feature quantities (area, center of gravity, major axis, minor axis, number of vertices, etc.) of pixel groups, and object recognition for recognizing an object using a learning model using machine learning or so-called deep learning.The reference data may be a reference image, a reference model, a reference feature quantity, or learning data regarding the object W or the characteristic part of the object W, and may be two-dimensional data or three-dimensional data. In each of the object detection techniques, the object detection means 21 calculates an evaluation value indicating the degree of similarity of the object W using a known evaluation function, and a position at which the evaluation value (i.e., the degree of similarity) is highest is detected, which is taken as the position of the object W. The evaluation value thus obtained is used for determination in the imaging condition calculation means 22, which will be described in more detail below.The evaluation function can use, for example, SSD (sum of absolute differences), SAD (sum of squared differences), NCC (normalized cross-correlation), ZNCC (zero-normalized cross-correlation), or the like. The evaluation function can also be a learning model formed by machine learning, or based on deep learning, etc. If there are multiple objects W or characteristic parts of the detected object W, it is preferable to append labels such as W1, W2, W3, etc. in the order of ascending evaluation values, after which the processing is performed.

[0028] If the calculated evaluation value is smaller than a predetermined threshold value, the imaging condition calculation device 22 can calculate, for example, the optimal imaging conditions of the time-of-flight sensor for the object W or the characteristic part of the object W for which the highest evaluation value is detected. The imaging conditions include at least one of the following parameters: integration time, light emission period of the reference light, luminous intensity of the reference light, required imaging repetitions, etc., as described above. The imaging condition calculation device 22 can calculate the imaging conditions based on an image of the luminous intensities, distances, etc. of the detected object W or a specific part of the object W, ie, based on an image of an imaging region specified as the object W or the characteristic part of the object W in the captured image.In other words, the object detection device 21 performs a calculation regarding the position and evaluation value of the object W or a characteristic part of the object W based on a first image taken under the predetermined imaging conditions and the predetermined imaging posture. If the calculated evaluation value is less than a predetermined threshold value, the imaging condition calculation device 22 calculates the imaging conditions. The object detection device 21 then, in turn, calculates the position and evaluation value of the object W or a characteristic part of the object W based on a second image taken under the calculated imaging conditions. (Method 1 for integration time calculation)

[0029] Fig. Figure 3 shows an example of a method for calculating the integration time. Generally, the magnitude σ of the distance measurement fluctuations of the time-of-flight sensor 10 can be expressed by the following equation. In the equation, s is the reflected intensity of the reference light at the object W or at the specific part of the object W, k is a stationary noise component of the time-of-flight sensor 10 or the reflected light intensity of external light (ambient light), etc. σ∝s+ks⋅f

[0030] The above equation shows that in order to obtain distance measurements with low fluctuation, it is beneficial to increase the reflected intensity s of the reference light. Since the integration time corresponds to the exposure time during which the shutter of a so-called "conventional camera" is open, the reflected intensity s of the reference light can be increased by increasing the integration time. In order to increase the reflected intensity s of the reference light so that saturation just does not occur, the imaging condition calculation device 22 can calculate an integration time IT2 based on the ratio of the reflected intensity s1 of the reference light at the detected object W or the specified part of the object W and the level s stat which saturation occurs, as exemplified in the following equation. In the equation, IT1 is the integration time before adjustment, and α is a predetermined margin. In the equation, α is a ratio to be added as a margin (tolerance) to the measured reflected intensity s1 of the reference light, where the margin may be a constant or it may also be determined using a function. In this case, the imaging condition changing device 23 changes the configuration of the time-of-flight sensor 10 to the calculated integration time IT2, and the object detecting device 21 detects the position of the object W based on the image output due to the changed integration time IT2.By adjusting the integration time IT2 based on the reflected intensity s1 of the reference light of the object W or the specific part of the object W, the distance measurement fluctuation with respect to the object W is reduced, thereby improving the accuracy of the position detection of the object W with the object detection device 21. IT2=ssts1(1+α)⋅IT1 (Method 2 for integration time calculation)

[0031] The Fig. 4 and Fig. 5 shows a modified example of the method for calculating the integration time. In addition to the reflected light of the reference light S, the time-of-flight sensor 10 also receives the reflected light of the external light source A, which is present at the edge of the measurement environment (also referred to as ambient light). If Fig. 5 If the integration time IT is set only by the reflected intensity s1 of the reference light, saturation may occur if the reflected intensity a1 of the external light is strong. If the integration time IT is extended, the reflected intensity a2 of the received external light will also be proportionally amplified, further increasing the saturation. Therefore, it is preferable to set the integration time IT taking into account the reflected intensity a1 of the external light. The reflected intensity a1 of the external light can be calculated, for example, using the following equation, or it can also be determined from the amount of charge obtained by imaging without emitting the reference light S (i.e., only by the reflected light of the external light source).The imaging condition calculation means 22 calculates the integration time IT2 taking into account the reflected intensity a1 of the external light source according to, for example, the following equation. In this case, the imaging condition changing means 23 changes the configuration of the time-of-flight sensor 10 to the calculated integration time IT2, and the object detection means 21 detects the position of the object W based on the image output according to the changed integration time IT2. Thus, it is possible to reduce the distance measurement fluctuations with respect to the object W without the occurrence of saturation by adjusting the integration time IT2 taking into account the reflected intensity a1 of the external light source from the object W or from a specific part of the object W. a=(Q1+Q2+Q3+Q4)−2s2 IT2=sst(s1+a1)(1+α)⋅IT1 (Method 3 for calculating the integration time)

[0032] The Fig. 6 and Fig. 7 shows another modified example of a method for calculating the integration time. When a change in the imaging position occurs, the integration time can be calculated by considering the distances d1, d2 to the object W or to a specific part of the object W. In principle, the reflected intensity s of the reference light S is inversely proportional to the square of the distance d to the object W or to the specific part of the object W, as given in the following equation.Thus, the reflected intensity s2 of the reflected light S2 of the reference light in the imaging position P2 near the object W increases according to the square of the ratio of the distance d1 to the object W or to a certain part of the object W in the imaging position P1 to the distance d2 to the object W or to the certain part of the object W in the imaging position P2 compared to the reflected intensity s1 of the reflected light S1 of the reference light at the imaging position P1.Therefore, in order to increase the reflected intensity s2 of the reflected light S2 of the reference light so that saturation does not occur when the time-of-flight sensor 10 gets closer to the object W or to the specific part of the object W, the imaging condition calculation device 22 may calculate the integration time IT2 by considering the ratio of the distance d1 to the object W or to the specific part of the object W in the imaging position before the change and the distance d2 to the object W or to the specific part of the object W in the changed imaging position, as exemplified by the following equation. The distance d2 can be calculated from a motion vector P of the distance d1 to the object W calculated based on the image taken from the imaging position P1 and the subsequent imaging position P2.For example, if the object W is a dark object and the imaging conditions are adjusted after the imaging position has been changed first, the distance measurement value of the object W output by the time-of-flight sensor 10 after moving to the imaging position P2 can be used as the distance d2. The imaging condition changing device 23 changes the configuration of the time-of-flight sensor 10 according to the calculated integration time IT2 in this way, and the object detection device 21 detects the position of the object W based on the image output with the changed integration time IT2. Thus, by adjusting the integration time IT2 while taking into account the distances d1, d2 to the object W or to a specific part of the object W, the distance measurement fluctuation with respect to the object W can be reduced without saturation occurring, even when the imaging position changes. s∝1d2 IT2=sst(s1⋅(d1d2)2+a1)(1+α)⋅IT1 (Method for calculating luminous intensity)

[0033] The Fig. The imaging condition calculator 22 shown in Figure 1 can calculate the luminous intensity of the reference light using the same approaches as the methods 1 to 3 explained above for calculating the integration time. Generally, the amount of received light of the reflected light of the external light source increases or decreases proportionally with the increase or decrease of the integration time, but the amount of received light of the reflected light of the external light source does not change by increasing or decreasing the luminous intensity. For example, the following equation is for calculation considering the reflected intensity a1 of the external light source. This equation is applicable to cases where the imaging position is further changed, and it corresponds to Equation 8.The case where the reflected intensity of the external light source is not considered is defined by a1 = 0, and the case where there is no change in the imaging position is defined by d1 = d2. In the equation, LI1 is the luminous intensity before adjustment, and the imaging condition changing means 23 changes the configuration of the time-of-flight sensor 10 to the calculated luminous intensity LI2, and the object detecting means 21 detects the position of the object W based on the image output at the changed luminous intensity LI2. The luminous intensity of the reference light can be adjusted according to the number of light-emitting elements as described above, or it can be adjusted by a voltage value or a current value applied to the light-emitting elements. By adjusting the luminous intensity based on the luminous intensity, distance, etc.of the object W or the specific part of the object W as described above, fluctuations in the distance measurement of the object W can be further reduced. LI2=sst−a1s1(1+α)⋅(d2d1)2⋅LI1 (Method for calculating the light emission period)

[0034] As Equation 3 shows, the magnitude σ of the distance measurement fluctuation of the time-of-flight sensor 10 can be reduced by increasing the light emission frequency f (=1 / T) of the reference light. The imaging condition calculation device 22 can further calculate the light emission frequency f (light emission period T; as always included below) of the reference light based on the distance d to the detected object W or to a specific part of the object W. However, it should be noted that when the light emission frequency f is increased, the distance measurement error due to aliasing occurs. Fig. Figure 8 shows an example of the occurrence of the distance measurement error due to aliasing, and the figure also explains a method for calculating a suitable light emission frequency f. For example, if the light emission frequency f of the reference light is 30 MHz (light emission period T 33 nm), then according to the following equation, the distance measurement range I is max, in which the time-of-flight sensor 10 performs a correct distance measurement, is 5 m. Therefore, if, for example, the actual distance d to the detected object W or to a specific part of the object W is 6 m, a distance measurement error occurs due to aliasing (the distance is erroneously measured as 1 m). Therefore, mapping is performed at a low light emission frequency with a value corresponding to an unknown distance to the object W or to the specific part of the object W. Once the object W or the specific part of the object W is detected and the distance is recognized, a suitable light emission frequency at which aliasing does not occur can be calculated.For example, if the distance d to the object W or to the specific part of the object W is 80 cm in imaging with a low emission frequency, in order to raise the light emission frequency f so that a distance measurement error due to aliasing does not occur, the imaging condition calculation means 22 calculates the light emission frequency f to be 150 MHz (light emission period T = 6 nm) so that the distance (for example, 1 m) obtained by adding a predetermined margin β (for example, 20 cm) to the detected distance d becomes the distance measurement range L. maxof the time-of-flight sensor 10, as can be seen, for example, from the following equation. In each case, the imaging condition changing device 23 changes the configuration of the time-of-flight sensor 10 to the calculated light emission frequency f, and the object detection device 21 detects the position of the object W based on the image output due to the changed light emission frequency f. Thus, by increasing the light emission frequency based on the distance d to the object W or to the specific part of the object W, the distance measurement fluctuation of the object W can be further reduced without causing distance measurement errors due to aliasing. Lmax=c2⋅f T=2(d+β)c

[0035] The distance measurement fluctuation of the time-of-flight sensor 10 according to Equation 3 is primarily due to so-called shot noise, and the distance measurement fluctuation exhibits a distribution that can be considered a normal distribution. Thus, a distance measurement value with reduced distance measurement fluctuation can be obtained by capturing multiple images from the same imaging position and processing them, such as averaging. Back to Fig. 1: If the evaluation value is still smaller than the predetermined threshold, the imaging condition calculation unit 22 may increase the number of required imaging repetitions (or the time filter constant). In this case, the imaging condition changing unit 23 changes the configuration of the time-of-flight sensor 10 according to the calculated required imaging repetitions (or the filter constant), and the object detection unit 21 detects the position of the object W based on the combined image from the plurality of images acquired according to the required number of imaging repetitions (or the time filter constant). In this way, by increasing the number of imaging repetitions (or the time filter constant), it is possible to further reduce the distance measurement fluctuations with respect to the object W.

[0036] If the calculated evaluation value is less than a predetermined threshold, the position / pose mapping calculation device 24 can calculate at least one mapping position. In this case, the robot 30 as the moving mechanism changes the position of the time-of-flight sensor 10 or the object W to the calculated mapping position, and the object detection device 21 detects the position of the object W based on the image output with the changed mapping position.

[0037] Fig. 9 shows an example of the image position calculation. Fig. 9, the object W is relatively far away from the time-of-flight sensor 10 at the imaging position P1, and the number of pixels with which the object W is captured (i.e., the number of distance measurement points) is small, and the reflected intensity of the reference light S from the object W is weak, so that the distance measurement fluctuation becomes relatively large. Since the irradiation range for the reference light S of the time-of-flight sensor is designed to be substantially the same as the imaging range according to the imaging position P1, the object W is irradiated with reference light S reflected by the highly reflective object O near the object W, so that a phenomenon called "multi-path" occurs, wherein the distance measurement value d1 for the object W is shifted backward, as indicated by the dashed line in the figure.Thus, in order to improve the evaluation value and detect accurate positions in the detection of the object W, the position / pose imaging calculation means 24 may calculate an imaging position at which the number of pixels for imaging the object W is as large as possible, an imaging position at which the reflected intensity of reference light S is increased, or an imaging position at which the influence of the so-called “multi-path” is reduced. (Method 1 for calculating the image position)

[0038] Since time-of-flight sensors (or TOF cameras) generally do not have a zoom function, the position / pose imaging calculator 24 according to the second embodiment calculates an imaging position P2, approaching close to the detected object W or the characteristic part of the object W so as to capture a larger image thereof. Fig. 10 shows an imaging position P2 in which a larger imaging can be performed by approaching the object W. Thus, an increase in the number of distance measurement points with respect to the object W or the characteristic part of the object W and a decrease in distance measurement fluctuations due to an increase in the reflected intensity of the reference light S from the object W can be expected. Since, in the imaging position P2, the highly reflective object O is not located within the pickup angle of the time-of-flight sensor 10 (that is, within the irradiation range of the reference light S), the object W is not affected by the multipath due to the highly reflective object O, and the phenomenon in which the distance measurement value d2 of the object W is shifted backward does not occur.By adjusting the object W or the characteristic (specific) part of the object W to the imaging position P2 such that the object is captured larger in the image, the position of the object W can be detected with higher accuracy.

[0039] The imaging position / posture calculation means 24 may calculate the imaging position and / or the imaging posture so that the detected object W or the characteristic part of the object W is captured in an imaging position and / or an imaging posture set in advance with reference to reference data. Fig. 11A shows an example of an imaging position and an imaging posture set in advance with respect to the object W. Generally, the imaging position and the imaging posture are set to be suitable for position detection of the object. When a reference image of the object W is used for the reference data, the imaging position and the imaging posture are used together when acquiring this image. There may be multiple imaging positions and imaging postures with respect to the object W, and in this case, a suitable imaging position and an appropriate imaging posture can be selected according to the state of the detected object. Fig. 11B shows an imaging position and imaging posture P3 set in advance for the object W. Since the object W has a relatively large distance and is inclined at the imaging position and imaging posture P1, the evaluation value is likely to be relatively low. As described above, when the evaluation value is low, the imaging position / posture calculation means 24 can calculate an appropriate imaging position and imaging posture based on the detected position of the object W, thereby achieving the imaging position and imaging posture P3 set in advance with respect to the reference data of the object W. As a result, better position detection of the object W is enabled. (Method 2 for image position calculation)

[0040] As in Fig. 9, bottom right, the imaging position / pose calculator 24 can specify a surface of the highly reflective object O that exerts the multi-path influence and calculate an imaging position P4 at which the surface of the object O is not irradiated with reference light from the time-of-flight sensor 10. Since, in time-of-flight sensors, in general, the reference light emission range and the reflected light reception range coincide and can thus be regarded as being at the same position, the imaging position at which the surface of the object O is not visible is the position at which the surface of the object O is not irradiated with reference light, so that the multi-path influence from the surface of the object O does not occur. Fig. 12A shows an imaging position P1 in which the surface F of the object O, which exerts a multi-path influence, is visible and Fig. 12B shows an imaging position P4 in which the surface F of the object O, which exerts a multi-path influence, is visible. The imaging position / pose calculation device 24 can specify the surface of the highly reflective object O in the vicinity of the detected object W or the characteristic part of the object W within the image, and further calculate the position and pose of the specified surface F and calculate a point on the extended surface corresponding to the surface F as the imaging position P4. Depending on the arrangement of the object and the highly reflective object, the time-of-flight sensor 10 can be located in an imaging position in which the surface of the highly reflective object is not visible by bringing the time-of-flight sensor 10 close to the object W, as in Fig. 9 top right and in Fig. 10, right side. Thus, by setting the imaging position P4, in which the surface of the highly reflective object O, which would exert a multipath influence, is not seen, the position of the object W can be detected with improved accuracy.

[0041] Fig. 13A shows an example of a movement mechanism, Fig. 13B shows a modified example of the movement mechanism. The movement mechanism may be a robot 30 that moves the runtime sensor 10 according to Fig. 13A, or it can be a robot that supports the object W, as in Fig. 13B. In the example according to Fig. 13A, the time-of-flight sensor 10 is mounted on the tip of the robot arm, and the robot 30, which supports the time-of-flight sensor 10, changes the position of the time-of-flight sensor 10 to the calculated imaging position, so that the position of the object W can be accurately detected. In the example according to Fig. 13B, the robot 30 supporting the object W detects the precise position, posture, size, etc. of the object W in the supported state using the time-of-flight sensor 10 at a predetermined fixed point based on approximate position information for the object W. The robot 30 supporting the object W changes the position of the object W to the calculated mapping position, so that the position, posture, size, etc. of the supported object O can be detected with higher accuracy. By configuring the movement mechanism in the form of the robot 30, it is possible to perform both picking operations and positioning operations performed by the robot 30, as well as object detection.

[0042] Fig. 14 shows the operation flow of the object detection system 1 according to the present embodiment. When the object detection system 1 begins the detection process, imaging is performed with the time-of-flight sensor under the specified imaging conditions and in the specified imaging position in step S10. The image output by the time-of-flight sensor is acquired in step S11. Once the required image repetitions (or a time filter constant) have been set, the combined image can be created in step S12. The combined image can be created on the time-of-flight sensor side or on the controller side (or a host computer, as described below).

[0043] In step S13, the position of the object is detected by calculating an evaluation value indicating the degree of similarity of the object based on the image output by the time-of-flight sensor and the reference data of the object or the characteristic part of the object stored in advance. If multiple objects are detected, markers such as W1, W2, W3, etc. can be assigned in the order of the highest evaluation values, and subsequent processing can be performed. Step S14 is not essential, but it is preferable to identify the surfaces of an object that may exert a multi-path influence on the object.

[0044] In step S15, it is determined whether the calculated evaluation value is smaller than a predetermined threshold. Furthermore, it can be determined whether the surface of an object that exerts a multi-path influence on the object is identified. If the evaluation value is smaller than the predetermined threshold in step S15, or if a surface of an object with a multi-path influence is identified (NO in step S15), the process proceeds to step S16. In step S16, the imaging conditions are calculated based on the light intensity, distance, etc. with respect to the object or the characteristic part of the object. If the evaluation value does not improve, the required image repetitions (or an appropriate time filter constant) can be applied.

[0045] In step S17, the imaging position is calculated based on the position (and, if applicable, the posture) of the object or the characteristic part of the object and the surface of the object that exerts a multi-path influence. At least one of steps S16 and S17 must be executed. In step S18, a change in the imaging conditions and / or a change in the imaging position is executed. The process then returns to step S10. In step S13, the position of the object is again detected by recalculating the evaluation value based on the now output image with the changed imaging condition and / or changed imaging position.

[0046] In step S15, if the evaluation value is again below the predetermined threshold or if a surface of an object with multi-path influence is again identified (NO in step S15), steps S16 to S14 are repeated. A limitation of the repetition loops can be provided so that when the loop has been repeated a predetermined number of times, a notification is issued and the detection process is aborted. If in step S15 the evaluation value is greater than the predetermined threshold and no surface exerting a multi-path influence is identified (YES in step S15), the detection process ends in step S19, and the detected position of the object W is the valid position. The operation of the robot is changed according to the valid position of the object W (not shown).

[0047] Fig.15 shows a modified example of the configuration of the object detection system 1. The object detection system 1 of this example may include a plurality of time-of-flight sensors 10, a host computer 50 connected wired or wirelessly to the plurality of time-of-flight sensors 10, a plurality of controllers 20 receiving commands from the host computer 50, a plurality of robots 30, and a plurality of tools 31 each controlled by the plurality of controllers 20. The host computer 50, such as the controllers 20, may be conventional computers or quantum computers. By incorporating essential components of the object detection system 1 into the host computer 50, the processing performance, maintenance, etc. of the object detection system can be improved.

[0048] With the embodiments described above, the position of the object W can be accurately detected by changing the imaging conditions and / or the imaging position based on the object image detected from the image of the time-of-flight sensor 10.

[0049] The above-described program executed by the processor may be recorded in a computer-readable, stable recording medium such as a CD-ROM or the like.

[0050] Although various embodiments have been described in detail above, the present invention is not limited to such examples and various modifications are possible within the scope of the following claims.

Claims

[1] Object detection system (1), comprising: a time-of-flight sensor (10) which outputs an image of a target space based on a phase difference between reference light emitted toward the target space and light reflected from the target space, an object detection device (21) which detects a position of an object (W) present in the target space on the basis of the output image, an imaging condition calculation device (22) which calculates imaging conditions based on an image of the detected object (W), including an integration time and / or a light emission period of the time-of-flight sensor (10), an imaging condition changing device (23) which changes a configuration of the time-of-flight sensor (10) according to the calculated imaging conditions, a position / posture image calculation device (24) which calculates at least one image position based on the image of the detected object (W), a movement mechanism (30) which changes a position of the time-of-flight sensor (10) and / or the object (W) to the calculated imaging position, wherein the object detection device (21) further calculates an evaluation value indicating the degree of similarity of the object (W) on the basis of the output image and pre-stored reference data relating to the object (W) or a characteristic part of the object (W), wherein, if the calculated evaluation value is smaller than a predetermined threshold value, the imaging condition calculation device (22) calculates the imaging conditions or the imaging position / posture calculation device (24) calculates the imaging position, and wherein the object detection device (21) detects a position of the object based on the image output according to the changed imaging position and / or the changed imaging conditions. [2] Object detection system (1), comprising: a time-of-flight sensor (10) which outputs an image of a target space based on a phase difference between reference light emitted toward the target space and light reflected from the target space, an object detection device (21) which detects a position of an object (W) present in the target space on the basis of the output image, an imaging condition calculation device (22) which calculates imaging conditions based on an image of the detected object (W), including an integration time and / or a light emission period of the time-of-flight sensor (10), an imaging condition changing device (23) which changes a configuration of the time-of-flight sensor (10) according to the calculated imaging conditions, a position / posture image calculation device (24) which calculates at least one image position based on the image of the detected object (W), a movement mechanism (30) which changes a position of the time-of-flight sensor (10) and / or the object (W) to the calculated imaging position, wherein the image calculation device (24) for position / pose calculates the image position such that the detected object (W) or a characteristic part of the object (W) is recorded in an enlarged manner in the image, and wherein the object detection device (21) detects a position of the object based on the image output according to the changed imaging position and / or the changed imaging conditions. [3] The object detection system (1) according to claim 2, wherein the object detection means (21) further calculates an evaluation value indicating the degree of similarity of the object (W) based on the output image and pre-stored reference data regarding the object (W) or a characteristic part of the object (W), wherein when the calculated evaluation value is smaller than a predetermined threshold value, the imaging condition calculation means (22) calculates the imaging conditions or the imaging position / posture calculation means (24) calculates the imaging position. [4] The object detection system (1) according to claim 1, wherein the image position / posture calculation means (24) calculates the image position such that the detected object (W) or a characteristic part of the object (W) is magnified in the image. [5] An object detection system according to any one of claims 1 to 4, wherein the imaging position / posture calculation means (24) calculates the imaging position and / or the imaging posture such that the detected object (W) or a characteristic part of the object (W) is captured in a predetermined imaging position and / or imaging posture with respect to the reference data of the object (W) or the characteristic part of the object (W). [6] The object detection system (1) according to any one of claims 1 to 5, wherein the position / posture imaging calculation means (24) identifies a surface of an object exerting a multi-path influence on the detected object (W) and calculates the imaging position such that the reference light emitted by the time-of-flight sensor (10) is not emitted onto the surface of the object. [7] The object detection system (1) according to any one of claims 1 to 6, wherein the moving mechanism (30) is a robot that supports the time-of-flight sensor (10) or a robot that supports the object (W). [8] The object detection system (1) according to any one of claims 1 to 7, wherein the imaging condition calculating means (22) calculates the integration time based on the reflected intensity of the reference light of the detected object (W) or a specific part of the object (W). [9] An object detection system (1) according to claim 8, wherein said imaging condition calculating means (22) calculates the integration time taking into account the reflected intensity of external light from the detected object (W) or from a specific part of the object (W). [10] An object detection system (1) according to claim 8 or 9, wherein the imaging condition calculating means (22) calculates the integration time taking into account the distance to the object (W) or to the specific part of the object (W). [11] The object detection system (1) according to any one of claims 1 to 10, wherein the imaging condition calculating means (22) calculates the light emission period based on a distance to the detected object (W) or to a specific part of the object (W). [12] The object detection system (1) according to any one of claims 1 to 11, wherein the imaging conditions further include a required frame rate of the time-of-flight sensor (10), and wherein the image is a combined image in which a plurality of images captured according to the required frame rate are combined. [13] An object detection system according to any one of claims 1 to 12, wherein the imaging conditions further include a light intensity of the time-of-flight sensor (10), and wherein the imaging condition calculation means (22) calculates the light intensity based on an image of the detected object (W).

Images