3D sensing device

Abstract

The three-dimensional sensing device (1) comprises a light irradiation unit (10) that forms a light irradiation area (100) extending in a first direction (Y) and moves the light irradiation area (100) in a second direction (X), an imaging unit (20) that performs an imaging operation to image a line imaging range (200) extending in the first direction (Y) and moves the line imaging range (200) in the second direction (X), and a control unit (30) that controls the light irradiation unit (10) and the imaging unit (20) so that a light curtain (300) is formed and controls the imaging unit (30). and a control unit (30) that detects an object (400) overlapping the light curtain (300) based on an image captured by the imaging unit (20), and the control unit (30) changes the position of the light curtain (300) in the depth direction (Z) corresponding to the distance from the imaging unit (20) by controlling the light irradiation direction of the light irradiation unit (10) and the line of sight direction of the imaging unit (20) and changes the thickness of the light curtain (300) in the depth direction (Z) by controlling the exposure time of the imaging unit (20).

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional sensing device for detecting objects present in a target space. [Background technology]

[0002] In recent years, a system has been proposed that includes a light source, a line sensor, a processor, and software for creating a light sheet plane (also called a "light curtain"), and detects objects in an area defined by the intersection of the light sheet plane and the sensing plane of the line sensor (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-530349 A (see, for example, claim 1) Summary of the Invention [Problem to be solved by the invention]

[0004] However, when detecting an object (e.g., acquiring a three-dimensional shape) using the above conventional system, it is necessary to repeatedly project the light sheet plane. For example, to obtain depth information of a range of 1 m in the depth direction with a resolution of 10 mm, it is necessary to project 100 light sheet planes at different positions in the depth direction. In this case, assuming that a 50 fps rolling shutter camera is used as the line sensor, it will take 2 seconds to obtain the depth information of one scene. This is a lower depth acquisition rate than the method using a ToF or stereo camera.

[0005] An object of the present disclosure is to provide a three-dimensional sensing device capable of speeding up the detection of objects present in a target space. [Means for solving the problem]

[0006] The three-dimensional sensing device of the present disclosure is A three-dimensional sensing device that detects an object existing in a target space, the three-dimensional sensing device comprising: a light irradiation unit that forms a light irradiation area extending in a first direction by irradiating light and moves the light irradiation area in a second direction that intersects with the first direction; an imaging unit that performs an imaging operation to image a line imaging range extending in the first direction and moves the line imaging range in the second direction; and a control unit that controls the light irradiation unit and the imaging unit so as to form a light curtain consisting of a planar imaging area that is an intersection range where the light irradiation area and the line imaging range intersect, and detects the object that overlaps with the light curtain based on an image captured by the imaging unit. the control unit changes a depth direction position of the light curtain corresponding to a distance from the imaging unit by controlling a light irradiation direction of the light irradiation unit and a line of sight direction of the imaging unit, changes a thickness of the light curtain in the depth direction by controlling an exposure time of the imaging unit, executes a search operation to detect a first detection position which is a position of the object in the depth direction while changing the position of the light curtain in the depth direction by a first interval, and executes a detection operation to detect a second detection position which is a position of the object in the depth direction while changing the position of the light curtain in the depth direction by a second interval narrower than the first interval within a range around the first detection position. The present invention is characterized by the above. Effect of the Invention

[0007] According to the device of the present disclosure, it is possible to speed up the detection of objects present in a space. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a perspective view showing a schematic configuration of a three-dimensional sensing device for performing light curtain sensing in the present disclosure. [Diagram 2] FIG. 1 illustrates an example of the configuration of a laser scanner. [Diagram 3] 3A to 3C show a 3D image of an object, an image of the object's background, and a composite image of these, respectively. [Figure 4] FIG. 13 is a diagram illustrating the generation of a light curtain in the XZ plane. [Diagram 5] 13A and 13B are diagrams illustrating the operation of a light curtain on a virtual screen. [Figure 6] 13A and 13B are diagrams illustrating the operation of a light curtain on a virtual screen. [Figure 7] 11A and 11B are diagrams illustrating a control method for alternately acquiring sensing by a light curtain and a background image. [Figure 8] FIG. 1 illustrates a geometric arrangement for generating a light curtain. [Figure 9] 1A to 1C are diagrams illustrating an example of the operation of a three-dimensional sensing device for measuring an object. [Figure 10] 1 is a schematic diagram showing a configuration of a three-dimensional sensing device according to a first embodiment. [Figure 11] FIG. 1A is a diagram showing how a three-dimensional sensing device according to embodiment 1 detects a pyramidal object, and FIG. 1B is a diagram showing the portion detected by the light curtain of FIG. [Figure 12] 4A to 4C are diagrams illustrating how the three-dimensional sensing device in accordance with the first embodiment detects a pyramidal object using light curtains at various positions in the depth direction. [Figure 13] 1A is a diagram showing the results of a three-dimensional sensing device of embodiment 1 detecting a pyramidal object using a light curtain at each position in the depth direction, and FIG. 1B is a diagram showing the shape of the object obtained by combining the results of FIG. 1A. [Figure 14] A diagram showing how a light curtain is formed by an imaging area, which is an intersection area where a moving light irradiation area formed by light irradiated from a light irradiation unit and a moving line imaging range formed by an imaging unit intersect. [Figure 15] FIG. 1A shows that the thickness of the light curtain increases when the exposure time of the imaging unit is increased, and FIG. 1B shows that the thickness of the light curtain decreases when the exposure time of the imaging unit is shortened. [Figure 16] FIG. 2A is a diagram showing an example of a searching operation of the three-dimensional sensing device according to the first embodiment, and FIG. 2B is a diagram showing an example of a detecting operation of the three-dimensional sensing device according to the first embodiment. [Figure 17] 5 is a flowchart showing an example of the operation of the three-dimensional sensing device according to the first embodiment. [Figure 18] 13A and 13B are diagrams showing another example of the search operation of the three-dimensional sensing device according to embodiment 1, and 13C is a diagram showing another example of the detection operation of the three-dimensional sensing device according to embodiment 1. [Figure 19] FIG. 1A is a diagram showing another example of the search operation of the three-dimensional sensing device according to the first embodiment, and FIG. 1B is a diagram showing another example of the detection operation of the three-dimensional sensing device according to the first embodiment. [Figure 20] 10 is a flowchart showing another example of the operation of the three-dimensional sensing device according to the first embodiment. [Figure 21] 10 is a flowchart showing another example of the operation of the three-dimensional sensing device according to the first embodiment. [Figure 22]5A and 5B are diagrams illustrating a searching operation of the three-dimensional sensing device according to the first embodiment. [Figure 23] 5A and 5B are diagrams illustrating a detection operation of the three-dimensional sensing device according to the first embodiment. [Figure 24] A figure showing how a light curtain is formed by an imaging area, which is an intersection area where a moving light irradiation area formed by light irradiated from a light irradiation unit in a three-dimensional sensing device of embodiment 2 intersects with a moving line imaging range formed by an imaging unit. [Diagram 25] FIG. 1A is a diagram showing a function θ(t) indicating the irradiation direction of the light irradiation unit, a function φ(t) indicating the line of sight direction of the imaging unit, and an exposure time; FIG. 1B is a diagram showing a camera trigger in the imaging unit; and FIG. 1C is a diagram showing the on / off state of the light irradiation unit. [Figure 26] 13A to 13C are diagrams showing how the thickness of the light curtain changes depending on the depth direction position of the light curtain and the exposure time of the imaging unit. [Figure 27] 13(A) to 13(H) are diagrams showing how the thickness of the light curtain changes depending on the depth direction position of the light curtain and the exposure time of the imaging unit. [Figure 28] 13A to 13C are diagrams showing how the thickness of the light curtain changes depending on the depth direction position of the light curtain and the exposure time of the imaging unit. [Figure 29] 13(A) to 13(D) are diagrams showing how the thickness of the light curtain changes depending on the depth direction position of the light curtain and the exposure time of the imaging unit. [Diagram 30] FIG. 1A is a diagram showing the position of the light curtain when the imaging unit is located downstream of the light irradiation unit in the direction of movement of the light irradiation area, and the direction of movement of the light irradiation area is clockwise; FIG. 1B is a diagram showing the position of the light curtain when the imaging unit is located downstream of the light irradiation unit in the direction of movement of the light irradiation area, and the direction of movement of the light irradiation area is counterclockwise. [Diagram 31]FIG. 1A is a diagram showing the position of the light curtain when the imaging unit is located upstream of the light irradiation unit in the direction of movement of the light irradiation area and the direction of movement of the light irradiation area is clockwise, and FIG. 1B is a diagram showing the position of the light curtain when the imaging unit is located upstream of the light irradiation unit in the direction of movement of the light irradiation area and the direction of movement of the light irradiation area is counterclockwise. [Diagram 32] 29(A) to (D) are diagrams showing an example (part 1) of a search operation in the case of FIG. 29(A). [Diagram 33] 29(A) to (C) are diagrams showing an example (part 2) of a search operation in the case of FIG. 29(A). [Diagram 34] 29(A) to (C) are diagrams showing another example of the search operation in the case of FIG. 29(A). [Diagram 35] FIG. 11 is a schematic diagram showing a configuration of a three-dimensional sensing device according to a third embodiment. [Diagram 36] 13(A) and 13(B) are diagrams illustrating an example of a searching operation of the three-dimensional sensing device according to the third embodiment. [Figure 37] FIG. 11 is a diagram illustrating an example of the operation of the three-dimensional sensing device according to the third embodiment. [Figure 38] FIG. 1A is a diagram showing the angular function θ(t) of the irradiation direction of the light irradiation unit and the angular function φ(t) of the line of sight direction of the imaging unit, FIG. 1B is a diagram showing the camera trigger in the imaging unit, and FIG. 1C is a diagram showing the on / off of the light irradiation unit. [Figure 39] FIG. 11 is a diagram illustrating an example of the operation of the three-dimensional sensing device according to the third embodiment. [Diagram 40] FIG. 1A is a diagram showing the angular function θ(t) of the irradiation direction of the light irradiation unit and the angular function φ(t) of the line of sight direction of the imaging unit, FIG. 1B is a diagram showing the camera trigger in the imaging unit, and FIG. 1C is a diagram showing the on / off of the light irradiation unit. [Diagram 41] 13A to 13D are diagrams illustrating an example of a searching operation of the three-dimensional sensing device according to the third embodiment. [Diagram 42]13(A) and 13(B) are diagrams illustrating an example of a searching operation of the three-dimensional sensing device according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A three-dimensional sensing device according to an embodiment will be described below with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified. Each figure shows the coordinate axes of an XYZ Cartesian coordinate system.

[0010] 1. Overview of the configuration and operation of the three-dimensional sensing device according to the first to third embodiments <3D sensing device 500> In the XYZ Cartesian coordinate system shown in each drawing, the Z direction (Z axis) indicates a direction perpendicular to the virtual screen, the Y direction (Y) indicates a direction perpendicular to the Z direction and parallel to the imaging area, and the X direction (X axis) indicates a direction perpendicular to both the Z direction and the Y direction. The configuration of the three-dimensional sensing device 500 is applicable to the three-dimensional sensing devices 1 to 3 according to the first to third embodiments.

[0011] Fig. 1 is a perspective view showing a schematic configuration of a three-dimensional sensing device 500 (also called a light curtain sensor device) for performing light curtain sensing in the present disclosure. In the example shown in Fig. 1, a state in which a light curtain 525 is generated by a combination of a laser scanner 501 and a camera 502 is shown.

[0012] The three-dimensional sensing device 500 has a laser scanner 501, a camera 502, and a synchronization circuit 504 as a control circuit. In the three-dimensional sensing device 500, the laser scanner 501 and the camera 502 are arranged side by side along the X direction, and are controlled by the synchronization circuit 504.

[0013] Laser scanner 501 emits a laser 512 (also called a line laser) that spreads in the Y direction. A virtual screen 510, which is a plane for the purpose of explanation, exists perpendicular to the Z direction at a position Z0 away from laser scanner 501 in the Z direction. The laser emitted from laser scanner 501 becomes linear laser 512 extending in the Y direction on virtual screen 510.

[0014] <Laser Scanner 501> 2 is a diagram showing an example of the configuration of a laser scanner 501. The laser scanner 501 emits a laser. The laser scanner 501 can scan in a direction parallel to the X-axis with a laser extending in a direction parallel to the Y-axis in an XYZ orthogonal coordinate system. The laser scanner 501 and the camera 502 are arranged along the X-axis in the XYZ orthogonal coordinate system.

[0015] A laser beam with a wavelength λ is emitted from a laser light source 505, and the laser beam is converted into a laser 512 (i.e., laser light) that spreads in the Y direction by a line beam generating element 506. The line beam generating element 506 is, for example, a cylindrical lens or a Powell lens. The laser 512 is deflected in the Z direction by a galvanometer mirror 507. The galvanometer mirror 507 can be swung around the Y axis at an angle of ±α, and the laser 512 is scanned around the Y axis within an angle range of ±2α. On the virtual screen 510, the laser 512 is scanned from the +X direction to the -X direction, and the area of ​​the scan range 513 in FIG. 1 or FIG. 2 is fully illuminated.

[0016] <Camera 502> The camera 502 can scan an imaging area 522 (also called a linear imaging area) extending in a direction parallel to the Y axis in an XYZ orthogonal coordinate system in a direction parallel to the X axis in the XYZ orthogonal coordinate system. For example, the camera 502 captures an image of an object that intersects with a two-dimensional surface in a three-dimensional space by scanning in the -X direction of the X axis.

[0017] In this embodiment, the camera 502 is a rolling shutter camera. The camera 502 reads out the luminance information of the image sensor pixels in order from the top row. By shortening the exposure time of the camera 502 under the control of the synchronization circuit 504, the linear imaging area extending in the row direction of the image sensor can be scanned in the column direction perpendicular to the row direction. For example, in a rolling shutter camera with 1000 rows that operates at a frame rate of 50 fps, if the exposure time per row is te=20 μs, adjacent rows can be separated in time and sensing can be performed row by row. In actual operation, the synchronization circuit 504 causes the camera 502 to operate such that the exposure area is shifted in order of five rows at a time, for example, with an exposure time of te=100 μs.

[0018] Normally, a rolling shutter camera is used so that the row direction is horizontal, but in this embodiment, the camera 502 is installed rotated -90° around the optical axis. The camera 502 is installed so that the optical axis direction of the camera 502 faces slightly inward (-X direction) from the Z direction so that the entire imaging range 523 of the camera 502 on the virtual screen 510 roughly overlaps with the entire laser scan range 513. However, even if the camera 502 faces the Z direction, if there is an overlap between the entire imaging range 523 and the entire scan range 513, a light curtain 525 is generated within the overlapping region.

[0019] In addition, camera 502 is equipped with bandpass filter 508. The bandpass filter 508 may be installed at the tip of the lens barrel of camera 502, or may be installed inside camera 502. Bandpass filter 508 is a wavelength filter that passes only light of wavelength λ emitted from laser light source 505.

[0020] <Synchronization circuit 504> The synchronization circuit 504 controls the laser scanner 501 and the camera 502. The synchronization circuit 504 can synchronize the operation of the laser scanner 501 and the operation of the camera 502. For example, the synchronization circuit 504 controls the laser scanner 501 and the camera 502 so that the irradiation direction of the laser emitted from the laser scanner 501 and the line of sight direction of the camera 502 intersect on a two-dimensional plane in a three-dimensional space.

[0021] 3(A) to 3(C) are diagrams showing a 3D image of an object, an image of the background of the object, and a composite image of these, respectively. In this embodiment, the three-dimensional sensing device 500 can generate a composite image by acquiring a 3D image of the object and an image of the background of the object and synthesizing these images. For example, as shown in FIG. 3(A), the synchronization circuit 504 acquires an image of the object by synchronizing the scan (also called the first scan) of the laser scanner 501 and the camera 502 in the -X direction. As shown in FIG. 3(B), the synchronization circuit 504 acquires an image of the background of the object by making the shutter open time (also called the exposure time) of the camera 502 longer than when the first scan in the -X direction is performed while the laser is irradiated. The image of the background of the object is also called a background image. The background image is acquired, for example, by the laser scanner 501 and the camera 502 scanning from the +X direction to the -X direction of the X axis.

[0022] The synchronization circuit 504 generates a composite image by combining the image of the object and the background image. As shown in Fig. 3(C), this process enables the three-dimensional sensing device 500 to generate a composite image in which the 3D image of the object and the background image are combined. In other words, the three-dimensional sensing device 500 is an image sensing device that can generate an image including an image of the sensing area and an image behind or around the sensing area.

[0023] <Light curtain sensing> FIG. 4 is a diagram for explaining the generation of light curtain 525 in the XZ plane. FIG. 5 and FIG. 6 are diagrams for explaining the operation of light curtain 525 on virtual screen 510. In FIG. 4, the angle that irradiation direction 511 of laser 512 from the Z direction makes with the Z axis is θ. Note that the clockwise direction is positive. In FIG. 4, angle θa between irradiation direction 511a and the Z axis at a certain time ta has a negative value. Similarly, angle θb between irradiation direction 511b and the Z axis at time tb has a positive value. Note also that the positive direction of the X axis is the left direction in the figure.

[0024] In FIG. 4, the camera 502 is installed rotated -90° around the optical axis, so that the line of sight 521 of the camera 502 scans clockwise in the XZ plane. The angle with the Z axis is φ clockwise. In FIG. 4, the angle φa between the line of sight 521a and the Z axis at time ta, and the angle φb between the line of sight 521b and the Z axis at time tb are both positive values. Here, at time ta, the irradiation direction 511a and the line of sight 521a intersect at the intersection 524a on the virtual screen 510a. Similarly, at time tb, the irradiation direction 511b and the line of sight 521b intersect at the intersection 524b on the virtual screen 510a. FIG. 5 and FIG. 6 show the relationship between the laser 512 and the imaging area 522 on the virtual screen 510a at the distance Za and the virtual screen 510b at the distance Zb.

[0025] On the virtual screen 510a, the laser 512 and the imaging area 522 are scanned so as to always overlap each other. As will be described later, such an operation is possible by appropriately controlling the angle of the galvanometer mirror 507 with respect to time. At this time, since the virtual screen 510b is installed at a different distance from the virtual screen 510a, as is clear from FIG. 6, the laser 512 and the imaging area 522 do not overlap each other.

[0026] In a state where light curtain 525 is generated on virtual screen 510a, when light curtain 525 intersects with an object (also referred to as object 541), an image outside the intersecting area is not captured by camera 502. This is because the exposure time of each pixel in one frame is extremely short (for example, exposure time te=60 μs), and further, light rays other than those of the laser wavelength are cut by bandpass filter 508 before they reach the image sensor inside camera 502. Therefore, the background is not captured by light curtain sensing.

[0027] Here, the conditions for scanning the laser 512 and the imaging area 522 so that they overlap on the virtual screen 510a will be described. First, when the laser 512 is linear and extends in the Y direction, the imaging area 522 of the camera 502 needs to extend in the Y direction accurately. For this purpose, a fine adjustment mechanism is provided for finely adjusting the attitude of the camera 502 around the optical axis. In addition, the imaging lens of the camera 502 needs to be a lens with distortion correction. Otherwise, the imaging area 522 will be distorted from a linear shape. In addition, in order to scan the laser 512 and the imaging area 522 while they overlap on the virtual screen 510a, it is necessary to synchronize the imaging start timing of the camera 502 and the scanning start timing of the laser scanner 501 with an appropriate shift, and for this purpose, a synchronization circuit 504 is used. Furthermore, the scanning speed of the laser 512 and the scanning speed of the imaging area 522 need to be the same on the virtual screen 510.

[0028] Fig. 7 is a diagram for explaining a control method for alternately acquiring sensing by a light curtain and a background image. In Fig. 7, the dashed line represents the angle function φ(t) of the line of sight direction of the camera 502 in the imaging area, and the solid line represents the angle function θ(t) of the irradiation direction 511 of the laser 512. In sections P1 and P3, a light curtain of a plane perpendicular to the Z axis is generated at a position at a different distance Z0, and background images are acquired in sections P2 and P4. Specific parameters used to calculate the graph shown in Fig. 7 will be described later.

[0029] First, the operation in section P1 will be described. Figure 8 is a diagram showing a geometric arrangement for generating a light curtain 525. An irradiation direction 511 of a laser 512 and a line of sight direction 521 of a camera 502 intersect on a virtual screen 510a placed at a distance Z0 in the Z direction. An angular function θ(t) of the irradiation direction 511 of the laser 512 and an angular function φ(t) of the line of sight direction 521 of the camera 502 are angles from the Z direction as shown in Figure 8. In the case of the arrangement in Figure 8, both the angle θ and the angle φ have positive values. The angle of the galvano mirror is half this angle (α=θ(t) / 2). The angular function φ(t) of the line of sight direction of the camera 502 is expressed by the following equation (1) using a constant coefficient k determined by the frame rate. φ(t)=k×t+φmin (1)

[0030] Since it is generally difficult to operate the line scanning speed of a rolling shutter camera at a speed other than a constant speed, the angular function θ(t) of the irradiation direction 511 of the laser 512 is controlled to match the angular function φ(t) of the line of sight direction of the camera 502. The relationship between θ(t) and φ(t) is obtained using FIG. 8. The rotation axis of the laser scanner 501 is set to point Q, the rotation axis of the line of sight direction 521 of the camera 502 is set to point R, and the distance between them is set to b. The foot of the perpendicular line drawn from the intersection point 524 to the line QR is set to point P. Then, the signed distance PR is expressed as Z0*tanφ using φ. In addition, since the signed distance PQ is expressed as Z0*tanθ, the following equation (2) holds. Z0×tanφ=Z0×tanθ+b (2)

[0031] In other words, when the angle φ is fixed, the angle θ of the laser 512 is calculated by the following equation (3). θ=Arctan(tanφ-b / Z0) (3)

[0032] When the virtual screen 510a is a plane perpendicular to the Z axis, the distance Z0 is a constant value. The function that generates the planar light curtain 525 at the distance Z0 is shown in the graph of section P1 in Figure 7. The specific parameters for plotting the angle function of section P1 are as follows:

[0033] <Parameters for Section P1> Scan range of laser 512: -25°≦θ≦25° (θmin=-25°, θmax=25°) Scan range of camera 502: -15°≦φ≦35° (φmin=-15°, φmax=35°) b=0.15m The time it takes for the camera 502 to scan the entire line is T=16 ms. Z0=0.5m

[0034] The coefficient k in equation (1) is expressed by the following equation (4). k = (φmax-φmin) / T (4)

[0035] Here, the camera 502 has a field angle of 50° in the XZ plane, and its optical axis is tilted at +10° with respect to the Z direction.

[0036] The position of the plane with parameters Z0=0.5m in section P1 corresponds to the position marked by virtual screen 510b in FIG. 4. In the graph of section P1 shown in FIG. 7, the angle function φ(t) of the line of sight of camera 502 changes with time t as a linear function as expressed by equation (1), and changes within the range of φmin=-15° to φmax=35° between time t=0 and 16 ms. The angle θ of laser 512 during this period is given by equation (3), and the angle function θ(t) of irradiation direction 511 of laser 512 in section P1(1) is shown. θ and φ each have a limited scanning range, and a light curtain is generated only when both θ and φ are within the scanning range. In other words, the generation area of ​​the light curtain is between point 526 and point 527 in FIG. 4.

[0037] The intersection point between the irradiation direction of the laser 512 at the minimum angle θmin and the virtual screen 510b is designated as 526, and the intersection point between the line of sight 521 of the camera 502 at the maximum angle φmax and the virtual screen 510b is designated as 527.

[0038] When the laser scan in section P1(1) ends, the angle of galvanometer mirror 507 must be returned in the negative direction in preparation for the next laser scan. Because galvanometer mirror 507 is an object with a moment of inertia, a non-small finite time Tback is required to return it to the start position of the next scan. This differs depending on the product specifications of galvanometer mirror 507, but for example, a time of Tback = 4 ms is required. In section P1(2), the angle of laser 512 is swung from +25° to -25° using a sine function.

[0039] <Section P3> In section P3, light curtain 525 is set at a position with a distance Z0=2 m different from that in section P1. The parameters other than Z0 are the same as those in section P1.

[0040] <Operation of the synchronous circuit> A synchronization circuit 504 is used to operate the angle function φ(t) of the line of sight direction of the camera 502 and the angle function θ(t) of the irradiation direction 511 of the laser 512 at the accurate timing as shown in Fig. 7. The synchronization circuit 504 is programmed in advance with a function waveform that gives the angle function θ(t) of the irradiation direction 511 of the laser 512 and a timing for outputting a trigger pulse shown in Fig. 7 in accordance with the angle function θ(t). An analog voltage waveform that gives the angle function θ(t) is output from the synchronization circuit 504 to the laser scanner 501. A trigger pulse is output to the camera 502. The camera 502 starts capturing one frame of image at the rising edge of the trigger pulse voltage. That is, it starts scanning the line capture area.

[0041] <3D measurement using light curtain sensors> In Figure 7, planar light curtains 525 were generated at positions Z0 = 0.5 m and 2 m in sections P1 and P3, respectively. However, if multiple light curtains are installed with narrower spacing in the Z direction on the light curtain surface, three-dimensional measurements can be performed.

[0042] Fig. 9 is a diagram showing an example of the operation of the three-dimensional sensing device that measures an object 541. In Fig. 9, in a section Pn, a light curtain is installed at a distance Zn calculated by the following formulas (5) and (6). n = 1, 2, . . . , N (N is a natural number) (5) Zn = Z0 + (n-1)ΔZ (6) (n is a natural number)

[0043] That is, the interval between adjacent light curtains is ΔZ. An object 541 is placed in the sensing area of ​​such a light curtain. Images of cross sections of the object 541 where light curtains 525a, 525b, 525c, ... are intersected are captured by the camera 502. The cross-sectional area of ​​the object 541 reflected by the light curtain at a distance Zn is at a distance Zn from the light curtain sensor. Since many cross-sectional images at different distances Zn are acquired, distance information of the surface of the object 541, i.e., a depth map, can be obtained.

[0044] 2. First embodiment FIG. 10 is a schematic diagram showing the configuration of a three-dimensional sensing device 1 according to the first embodiment. The three-dimensional sensing device 1 is an image sensing device that projects an object present on the surface of a curtain-shaped light curtain and does not project an object not present on the surface of the light curtain. The three-dimensional sensing device 1 can also detect an object 400 present in a target space by moving the light curtain. "Detecting the object 400" means detecting the presence or absence of the object 400 or detecting the three-dimensional shape of the object 400, or both of these. For example, the three-dimensional sensing device 1 can detect whether or not there is a person as an object in a room as a target space by moving the light curtain. In addition, the three-dimensional sensing device 1 can detect the three-dimensional shapes of objects, equipment, etc. as objects present in the target space by moving the light curtain. The three-dimensional sensing device 1 has a light irradiation unit 10 as an illumination device, an imaging unit 20 as an imaging device, and a control unit 30 as a control device.

[0045] The light irradiation unit 10 is a device that forms a light irradiation region 100 extending in a first direction by irradiating light, and moves the light irradiation region 100 in a second direction that intersects with the first direction. In other words, the light irradiation unit 10 is a light scanning device (i.e., a laser scanner) that scans light (e.g., laser light) in a first direction and a second direction. The first direction is, for example, the Y direction, which is the height direction in FIG. 10 (or the extension direction of the light irradiation region 100). The second direction is, for example, the X direction, which is the horizontal direction in FIG. 10 (or the moving direction D100 of the light irradiation region 100). Although FIG. 10 shows a case where the first direction and the second direction are orthogonal to each other, the first direction and the second direction do not necessarily have to be orthogonal to each other. The light irradiation unit 10 is, for example, composed of a laser light source as a light source and a galvanometer mirror (i.e., a galvanometer scanner) as a light scanning optical system (scanning mirror). However, the configuration of the light irradiation unit 10 is not limited to this.

[0046] The imaging unit 20 is a device that performs an imaging operation to capture an image of a line imaging range 200, which is a linear imaging range extending in a first direction, and moves the line imaging range 200 in a second direction. In other words, the imaging unit 20 is a device that moves the line imaging range 200 in the second direction every time imaging is performed in the line imaging range 200. The first direction is the extension direction of the line imaging range 200 in FIG. 10, which is the same direction as the extension direction of the light irradiation area 100. The second direction is the movement direction D200 of the line imaging range 200, which is the same direction as the movement direction D100 of the light irradiation area 100. The imaging unit 20 is a sensor in which a plurality of imaging pixels are arranged in a line or two-dimensionally. The imaging unit 20 is, for example, a line sensor or a camera (for example, a rolling shutter camera).

[0047] The control unit 30 controls the light irradiation unit 10 and the imaging unit 20 so that a light curtain 300 consisting of an imaging area 80, which is an intersection area where the light irradiation area 100 and the line imaging range 200 intersect, is formed, and detects an object (for example, an object 400 in FIG. 11 described later) that overlaps the light curtain 300 based on an image captured by the imaging unit 20. The light curtain 300 is a collection of imaging areas 80 and is also called a "projection type light curtain". The control unit 30 can make the light curtain 300 a flat surface or a curved surface. The control unit 30 changes the position of the light curtain 300 in the depth direction (i.e., the depth direction) corresponding to the distance from the imaging unit 20 by controlling the light irradiation direction of the light irradiation unit 10 and the line of sight direction of the imaging unit 20 (i.e., the shooting direction). In FIG. 10, the depth direction is the Z direction. The control unit 30 can also change the thickness of the light curtain 300 in the depth direction by controlling the exposure time of the imaging unit 20 (i.e., the opening time of the camera shutter). The control unit 30 is composed of, for example, a control circuit. The control circuit may have a memory as a storage device that stores a program as software, and a processor such as a CPU (Central Processing Unit) that executes the program. The control unit 30 can also be composed of a dedicated processing circuit or a general-purpose computer.

[0048] Fig. 11(A) is a diagram showing how the three-dimensional sensing device 1 detects a quadrangular pyramid-shaped object 400. Fig. 11(B) is a diagram showing a portion 310 detected by the light curtain 300 of Fig. 11(A). As shown in Fig. 11(A), the light curtain 300 of the three-dimensional sensing device 1 has a thickness Tc. The three-dimensional sensing device 1 detects a portion 310 of the object 400 within an imaging region that is an intersection range that intersects with the light curtain 300.

[0049] FIG. 12 is a diagram showing how the three-dimensional sensing device 1 detects a pyramidal object 400 by light curtains 300a to 300e moving in the depth direction. The light curtains 300a to 300e shown in FIG. 12 show how the light curtain 300 in FIG. 11 moves in the depth direction. FIG. 12 shows an example in which the shape of the object 400 intersecting the light curtain 300a is detected, then the shape of the object 400 intersecting the light curtain 300b is detected, and then the shapes of the object 400 intersecting the light curtains 300c, 300d, and 300e are detected in order. FIG. 11(A) shows that the control unit 30 can change the thickness Tc of the light curtain 300 by adjusting the exposure time te of the imaging unit 20. FIG. 12 shows that the control unit 30 can change the position of the light curtain 300 in the depth direction by controlling the light irradiation unit 10 and the imaging unit 20.

[0050] Fig. 13(A) is a diagram showing the results of detection by the three-dimensional sensing device 1 of a pyramidal object 400 using a light curtain moving in the depth direction, and Fig. 13(B) is a diagram showing the shape of the object obtained by integrating the results of Fig. 13(A). Fig. 13(A) shows the results of detection using the light curtains 300a, 300c, 300e, etc. of Fig. 12 in a front view and a perspective view. Fig. 13(B) shows the integrated detection result 320.

[0051] 14 is a diagram showing how a light curtain 300 is formed by an imaging area 80, which is an intersection area of ​​a light irradiation area 100 formed by a laser beam, which is light irradiated from the light irradiation unit 10, and moving in the X direction, and a line imaging range 200 formed by the imaging unit 20 and moving in the X direction. As shown in FIG. 14, during an exposure time te (i.e., a shutter open time) when the imaging unit 20 captures an image of one line of a plurality of imaging pixels arranged in the Y direction, the light irradiation area 100 is scanned, and a light curtain 300 having a thickness Tc is formed in a portion where the moving light irradiation area 100 and the line imaging range 200 overlap. In the example of FIG. 14, the light irradiation unit 10 and the imaging unit 20 are arranged on a straight line extending in the X direction with an interval of a base length b.

[0052] Fig. 15(A) is a diagram showing that the thickness Tc of the light curtain 300 increases by lengthening the exposure time te of the imaging unit 20. Fig. 15(B) is a diagram showing that the thickness of the light curtain decreases by shortening the exposure time te of the imaging unit 20. Figs. 15(A) and (B) show that the thickness Tc of the light curtain 300 can be changed by the control unit 30 adjusting the exposure time te of the imaging unit 20.

[0053] Fig. 16(A) is a diagram showing an example of a searching operation of the three-dimensional sensing device 1, and Fig. 16(B) is a diagram showing an example of a detecting operation of the three-dimensional sensing device 1. Also, Fig. 17 is a flowchart showing an example of the operation of the three-dimensional sensing device 1.

[0054] First, as shown in Fig. 16(A), in a search operation, the control unit 30 sets the depth direction thickness of the light curtain 300 to a first thickness Tc1, and detects a first detection position, which is the depth direction position of the object 400 (step S11). In Fig. 16(A), the position of the light curtain 300 having the first thickness Tc1 is changed to four positions. Since the first detection position is an approximate position with low accuracy, a second detection position, which is a position of the object 400 with high accuracy, is obtained by the next detection operation.

[0055] When the first detection position, which is the position of the object 400, is detected in the search operation (YES in step S12), as shown in FIG. 16(B), in the detection operation, the control unit 30 changes the thickness of the light curtain 300 to a second thickness Tc2, which is thinner than the first thickness Tc1, in a range (for example, a predetermined range) around the first detection position, and detects the second detection position, which is the position in the depth direction of the object 400 (step S13). Since the second detection position is a position with high accuracy, it is also possible to detect the shape of the object 400. Note that the first thickness Tc1 may be the entire area of ​​the target space, and in this case, it is possible to detect whether or not an object is present in the target space. In addition, the search operation can be performed multiple times using light curtains with different thicknesses.

[0056] If the first detection position, which is the position of the object 400, is not detected in the search operation (NO in step S12), in the detection operation, the control unit 30 changes the thickness of the light curtain 300 to a second thickness Tc2 thinner than the first thickness Tc1 throughout the target space, and detects the second detection position, which is the position in the depth direction of the object 400 (step S14). In addition, the detection operation can be performed multiple times using light curtains with different detection densities (intervals) or different thicknesses Tc.

[0057] Figures 18(A) and (B) are diagrams showing another example of the search operation of the three-dimensional sensing device 1, and Figure 18(C) is a diagram showing another example of the detection operation of the three-dimensional sensing device 1. In Figures 16(A) and (B), the thickness of the light curtain in the search operation is of one type, but as shown in Figures 18(A) and (B), the thickness of the light curtain 300 in the search operation may be of two stages.

[0058] Fig. 19(A) is a diagram showing another example of the searching operation of the three-dimensional sensing device 1, Fig. 19(B) is a diagram showing another example of the detecting operation of the three-dimensional sensing device 1. Fig. 20 is a flowchart showing another example of the operation of the three-dimensional sensing device 1.

[0059] First, as shown in FIG. 19(A), in the search operation, the control unit 30 detects a first detection position, which is a position in the depth direction of the object 400, while changing the position in the depth direction of the light curtain 300 at a first interval I1 (step S21). FIG. 19(A) shows an example in which the position of the light curtain 300 is changed to four positions. Since the first detection position is an approximate position with low accuracy, a second detection position, which is a position of the object 400 with high accuracy, is obtained by the next detection operation. In addition, the first interval I1 does not need to be constant throughout the target space, and for example, the first interval I1 may be changed according to the distance from the light irradiation unit 10 or the imaging unit 20.

[0060] When a first detection position, which is the position of the object 400, is detected in the search operation (YES in step S22), as shown in FIG. 19(B), in the detection operation, the control unit 30 changes the light curtain 300 to a second interval I2 narrower than the first interval I1 in a narrow range around the first detection position, and detects the second detection position, which is the position in the depth direction of the object 400 (step S23). Since the second detection position is a position with high accuracy, it is also possible to detect the shape of the object 400. In addition, the second interval I2 does not need to be constant throughout the target space, and may be changed depending on the distance from the light irradiation unit 10 or the imaging unit 20, for example.

[0061] If the first detection position, which is the position of the object 400, is not detected in the search operation (NO in step S22), in the detection operation, the control unit 30 changes the light curtain 300 to a second interval I2 narrower than the first interval I1 throughout the target space, and detects the second detection position, which is the depth direction position of the object 400 (step S24).

[0062] Fig. 21 is a flowchart showing another example of the operation of the three-dimensional sensing device 1. First, as shown in Fig. 19(A), in a search operation, the control unit 30 sets the depth-direction thickness of the light curtain 300 to a first thickness Tc1, and detects a first detection position, which is the depth-direction position of the object 400, while changing the depth-direction position of the light curtain 300 at a first interval I1 (step S31). Since the first detection position is an approximate position with low accuracy, a second detection position, which is a position of the object 400 with high accuracy, is obtained by the next detection operation.

[0063] 19B, in the detection operation, when the first detection position, which is the position of the object 400, is detected (YES in step S32), the control unit 30 sets the depth direction thickness of the light curtain 300 to a second thickness Tc2 thinner than the first thickness Tc1 in a narrow range (for example, a predetermined range) around the first detection position, changes the light curtain 300 to a second interval I2 narrower than the first interval I1, and detects the second detection position, which is the position of the object 400 in the depth direction (step S33). Since the second detection position is a position with high accuracy, it is also possible to detect the shape of the object 400.

[0064] If the first detection position, which is the position of the object 400, is not detected in the search operation (NO in step S32), in the detection operation, the control unit 30 sets the depth thickness of the light curtain 300 to a second thickness Tc2 thinner than the first thickness Tc1 throughout the entire target space, changes the light curtain 300 to a second spacing I2 narrower than the first spacing I1, and detects the second detection position, which is the depth position of the object 400 (step S34).

[0065] 22(A) and (B) are diagrams illustrating a searching operation of the three-dimensional sensing device 1 according to embodiment 1. Fig. 23(A) and (B) are diagrams illustrating a detecting operation of the three-dimensional sensing device 1 according to embodiment 1.

[0066] As shown in Figures 22(A) and (B), the control unit 30 sets the width of the light curtain 300 in the second direction to a first width W1, performs a search operation, detects the position of the object in the second direction, and then, as shown in Figures 23(A) and (B), sets the width of the light curtain 300 in the second direction to a second width W2 narrower than the first width W1 in a range (e.g., a predetermined narrow range) around the position of the object in the second direction in the width direction, and performs a detection operation. Note that the processes in Figures 22 and 23 may be combined with the operations described in Figures 16 to 21.

[0067] Light curtain 300 is not a complete surface but has a finite thickness Tc. Light curtain 300, which is the imaging area per line of imaging pixels, is made up of a parallelogram area where the sectorial range of the line imaging area and the sectorial range irradiated by the laser during the exposure time intersect, and the thickness of the light curtain depends on the width of the laser light and the width of the line imaging area, the baseline length between the laser and the camera, the distance between the curtain, and the exposure time, as shown in Fig. 14.

[0068] In the first embodiment, the thickness of the light curtain 300 is dynamically changed by changing the parameter of the exposure time te, which can be easily changed while the light curtain 300 is in operation. As described in Figs. 15(A) and (B), the light curtain 300 becomes thicker when the exposure time te is increased, and the light curtain 300 becomes thinner when the exposure time te is decreased. When searching for an object in a target space (i.e., a scene), as shown in Fig. 16(A), the light curtain 300 is made thicker to widen the search range in the depth direction by projecting the light curtain 300 once or several times. This makes it possible to search for objects in a scene without overlooking them with a small number of projections.

[0069] When acquiring the three-dimensional shape of the object 400 by the three-dimensional sensing device 1, it is necessary to repeatedly project the light curtain 300 onto the scene. For example, to acquire depth information of a scene within a range of 1 m in the Z direction with a resolution of 10 mm, 100 planar light curtains are projected in the depth direction. In this case, if a rolling shutter camera with 50 fps is used, it takes 2 seconds to acquire depth information of one scene. This is a much lower depth acquisition rate than methods such as ToF (Time of Flight) and stereo cameras. In order to speed up the acquisition of the three-dimensional shape by the light curtain 300, it is effective to sparsely scan the scene with the light curtain 300 to search for the object, identify its position, and then perform limited high-density scanning of only the object and its surroundings.

[0070] The three-dimensional sensing device 1 of embodiment 1 has a function of changing the thickness Tc of the light curtain 300, and by providing a search operation, the total number of scans is reduced, after which a detection operation can be performed to obtain the three-dimensional shape of the object.

[0071] Furthermore, during the search operation, it is not necessary to scan the light curtain 300 parallel to the X-axis, and the curtain surface may be tilted to search a range that will not be overlooked.

[0072] The scanning start / stop distance in the 3D shape acquisition mode (i.e., during detection operation) is desirably set to, for example, the range enclosed by the nearest light curtains where nothing is detected before or after the position where an object is detected in the object search mode.

[0073] In addition, as shown in Figures 22 and 23, the ROI (Region Of Interest) setting of the rolling shutter camera as the imaging unit 20 is changed according to the rough position of the object in the X direction, thereby limiting the range of the light curtain 300 in the X direction, thereby making it possible to further speed up the acquisition of the 3D shape.

[0074] 3. Second embodiment In embodiment 2, we will explain a first case (Figures 26, 27, 30, and 32 to 34) in which the position of the imaging unit 20 is downstream of the position of the light irradiation unit 10 in the movement direction D100 of the light irradiation area 100, and a second case (Figures 28, 29, and 31) in which the position of the imaging unit 20 is upstream of the position of the light irradiation unit 10 in the movement direction D100 of the light irradiation area 100.

[0075] Figure 24 is a schematic diagram showing the configuration of the three-dimensional sensing device 2, and shows how a light curtain 300 of thickness Tc is formed by an imaging area 80, which is the intersection area where a moving light irradiation area 100 formed by light irradiated from the light irradiation unit 10 intersects with a moving line imaging range 200 formed by the imaging unit 20.

[0076] In the above description using FIG. 1 to FIG. 9, it was assumed that the irradiation direction (i.e., projection direction) of the laser light, which is the light irradiated from the light irradiation unit 10, and the line of sight direction (i.e., imaging direction) of the imaging unit 20 are straight lines having no width. However, in reality, the laser light is continuously scanned according to an angle function θ(t) which is a function of time t, and the irradiation direction of the laser light moves from irradiation direction 81 to 82 during the exposure time te of the imaging unit 20. While the irradiation direction of the laser light moves from irradiation direction 81 to 82, the area where the laser light intersects with the line imaging range 200, which is the imaging range by one row of imaging pixels of the imaging unit 20 having an angle of view 83, becomes the imaging area of ​​one row of imaging pixels of the imaging unit 20 (i.e., the detection area detected by one row of imaging pixels) 80. At this time, the width of the laser light also affects the imaging area 80. Therefore, a case in which the width of the laser light is taken into consideration in the case of FIG. 1 to FIG. 9 will be described.

[0077] Figure 25(A) is a diagram showing a function θ(t) indicating the irradiation direction of the light irradiation unit 10 of the three-dimensional sensing device 2 of embodiment 2, a function φ(t) representing the relationship between the line of sight of the imaging unit 20 and the exposure start time of the corresponding row, and an exposure time te, Figure 25(B) is a diagram showing the camera trigger in the imaging unit 20 of the three-dimensional sensing device 2, and Figure 25(C) is a diagram showing the on / off of the laser light output of the light irradiation unit 10.

[0078] 26(A) to (C) are diagrams showing how the thickness of the imaging region 80 forming the light curtain 300 changes depending on the depth direction position of the imaging region 80 forming the light curtain 300 and the exposure time te of the imaging unit 20. In Figs. 26(A) to (C), for the sake of simplicity, the angle of view 83 of one row of imaging pixels of the imaging unit 20 and the width 85 of the laser light are not taken into consideration.

[0079] 26(A) to 26(C), when a light curtain including a plane 13 perpendicular to the Z axis is created at a position that is a distance Z0 from the light irradiation unit 10 or the imaging unit 20, the value of the angular function of the irradiation direction of the laser light 11 from the light irradiation unit 10 is θ(t1) for the value of the angular function φ(t1) of the line of sight direction of the imaging unit 20 at a certain time t1. Here, if the exposure start time of the row of the imaging pixel of the imaging unit 20 that corresponds to the value of the angular function φ(t1) of the line of sight direction of the imaging unit 20 is set to t1, at time t1, the line of sight direction of the imaging unit 20 and the irradiation direction of the laser light 11 intersect at intersection 260 at the position of distance Z0, but the intersection during the exposure time te after time t1 is shifted from the position of distance Z0. 26(A) to (C), when the imaging unit 20 is on the +X side with respect to the light irradiation unit 10, and the scanning direction of one row of imaging pixels of the rolling shutter, which is the imaging unit 20, is clockwise (the scanning direction of the light irradiation unit 10 is also necessarily clockwise) (i.e., the first case), the imaging area is on the -Z side with respect to the position of the distance Z0. At the final time t1+te of the exposure, the laser light 12 has rotated Δθ=θ(t1+te)-θ(t1) from the laser light 11 at time t1, and captures the intersection 261. Here, the coordinates (x, z) of the intersection 261 can be calculated by the following formulas (7) and (8).

[0080]

number

[0081] The area connecting the intersection 260 and the intersection 261 is the imaging area of ​​one row of imaging pixels facing the line of sight direction 21. When the exposure time te is increased, Δθ becomes larger and the imaging area 80 expands toward the light irradiation unit 10 and the imaging unit 20 side, as shown in Figures 26(B) and 26(C).

[0082] The imaging area of ​​the imaging pixels in each row of the imaging unit 20 can be calculated from the angular function φ(t) of the line of sight direction of the imaging unit 20, the angular function θ(t) indicating the projection direction of the laser light, and the exposure time te. By calculating the imaging areas of the imaging pixels in all rows, the imaging area 80 that forms the light curtain 300 can be calculated.

[0083] 27(A) to (H) are diagrams showing how the thickness of the light curtain 300 (i.e., the thickness of the imaging area) changes depending on the position in the depth direction of the light curtain 300 and the exposure time te of the imaging unit 20. Figures 27(A) to (H) show the results of calculating the imaging area 80 of the light curtain 300 when the exposure time te is set to 0.1 ms, 0.5 ms, 1.0 ms, and 2.0 ms under the measurement conditions of the following parameters.

[0084] Scanning range of the light irradiation unit 10: -25°≦θ≦25° (θmin=-25°, θmax=25°) Scanning range of the imaging unit 20: -35°≦φ≦15° (φmin=-35°, φmax=15°) ·Baseline length b=0.15m Time required for the imaging unit 20 to complete scanning all lines of imaging pixels: T = 16 ms Distance Z0 = 1.0 m (Figures 27(A)-(D)), 2.0 m (Figures 27(E)-(H))

[0085] As shown in Fig. 27(A) to (D), when the distance Z0=1.0m, the imaging area in the Z direction is about 40mm when the exposure time is 0.1ms, but it is about 150mm when the exposure time is 0.5ms, about 300mm when the exposure time is 1.0ms, and about 450mm when the exposure time is 2.0ms. Also, as shown in Fig. 27(E) to (H), when the distance Z0=2.0m, the thickness, which is the range in the Z direction, increases compared to when Z0=1.0m. In this way, it is possible to control the Z direction range of the light curtain 300 by controlling the exposure time te. The boundary in the Z direction on the far side from the sensor of the imaging area is a straight line at the position of the distance Z0, but the boundary in the Z direction on the near side of the sensor is curved in a shape convex in the +Z direction.

[0086] 28(A) to (C) are diagrams showing how the thickness of the light curtain 300 changes depending on the position of the light curtain 300 in the depth direction and the exposure time te of the imaging unit 20. FIG. 28(A) to (C) show a case where the arrangement of the light irradiation unit 10 and the imaging unit 20 is inverted with respect to the Z axis (however, the scanning direction of the light irradiation unit 10 is clockwise). As shown in FIG. 28(A) to (C), an intersection 261 of the irradiation direction of the laser light 12 and the line of sight direction 21 of the imaging unit 20 at the exposure final time t1+te exists in the +Z direction with respect to the position of the distance Z0, so that the imaging area of ​​the imaging unit 20 has an imaging area 80 on the +Z side with respect to the position of the distance Z0.

[0087] 29(A) to (D) are diagrams showing how the thickness of the light curtain 300 (i.e., the thickness of the imaging area) changes depending on the position in the depth direction of the light curtain 300 and the exposure time te of the imaging unit 20. Figures 29(A) to (D) show the results of calculating the imaging area 80 of the light curtain 300 when the exposure time is set to 0.1 ms, 0.5 ms, 1.0 ms, and 2.0 ms under the measurement conditions of the following parameters.

[0088] Scanning range of the light irradiation unit 10: -25°≦θ≦25° (θmin=-25°, θmax=25°) Scanning range of the imaging unit 20: -15°≦φ≦35° (φmin=-15°, φmax=35°) ·Baseline length b=-0.15m Time required for the imaging unit 20 to scan all lines: T = 16 ms ·Distance Z0=1.0m

[0089] The thickness of the imaging region 80 of the light curtain 300 increases with each increase in the exposure time te. The boundary in the Z direction of the imaging region 80 on the side closer to the imaging unit 20 becomes a straight line at the position of the distance Z0, but the boundary in the Z direction on the side farther from the imaging unit 20 is curved in a concave shape in the +Z direction. In particular, the boundary on the +X side is greatly curved, and as shown in Fig. 29(D), when the exposure time is 2.0 ms, the imaging region 80 diverges.

[0090] Whether the Z-direction range of the imaging region 80 of the light curtain 300 extends in the +Z direction or the -Z side with respect to the position of the distance Z0 can be changed not only by the positional relationship between the light irradiation unit 10 and the imaging unit 20 but also by the scanning direction of the imaging unit 20. Here, the scanning direction of the light irradiation unit 10 is, in principle, the same as the scanning direction of the imaging unit 20.

[0091] Fig. 30(A) is a diagram showing the position of the imaging region 80 of the light curtain 300 in a first case in which the position of the imaging unit 20 is downstream of the position of the light irradiation unit 10 in the movement direction D100 of the light irradiation region 100 and the movement direction of the light irradiation region 100 is clockwise. Fig. 30(B) is a diagram showing the position of the imaging region 80 of the light curtain 300 in a first case in which the position of the imaging unit 20 is downstream of the position of the light irradiation unit 10 in the movement direction D100 of the light irradiation region 100 and the movement direction of the light irradiation region 100 is counterclockwise. In this case, the extension direction of the imaging region 80 of the light curtain 300 is the -Z side.

[0092] Fig. 31(A) is a diagram showing the position of the imaging region 80 of the light curtain 300 in a second case in which the position of the imaging unit 20 is upstream of the position of the light irradiation unit 10 in the movement direction D100 of the light irradiation region 100 and the movement direction of the light irradiation region 100 is clockwise. Fig. 31(B) is a diagram showing the position of the imaging region 80 of the light curtain 300 in a second case in which the position of the imaging unit 20 is upstream of the position of the light irradiation unit 10 in the movement direction D100 of the light irradiation region 100 and the movement direction of the light irradiation region 100 is counterclockwise. In this case, the extension direction of the imaging region 80 of the light curtain 300 is the +Z side.

[0093] When the imaging area 80 constituting the light curtain 300 is expanded in the +Z direction and used for monitoring purposes as a safety sensor, for example, the boundary at the position of the distance Z0 is easy to handle because it is a straight line, but it is highly likely that it will be difficult to accurately match the curved boundary side with the monitoring range. Therefore, in the case of a condition in which the extension direction of the light curtain 300 is in the +Z direction, as calculated in Figures 29(A) to (D), if the exposure time is increased and the thickness is increased, the light curtain 300 will be curved asymmetrically with respect to the X axis, and part of the light curtain 300 may diverge, which increases the constraints on control of the imaging area. Therefore, it is more preferable to control under conditions in which the extension direction of the imaging area 80 of the light curtain 300 is on the -Z side.

[0094] 32(A) to (D) are diagrams showing an example (part 1) of the search operation in the case of FIG. 30(A). In order to speed up the 3D measurement by the 3D sensing device 2, the rough position of the object 400 is first grasped by controlling the thickness of the light curtain 300 by the exposure time te. For example, the 3D shape of the object 400 in a range of 2 m in the Z direction at a distance of 0.5 m to 2.5 m is acquired. Here, a case where the object is present near Z=1.6 m is considered. In order to reduce the number of scans, the following two light curtains 300 are first used to divide the range of 0.5 to 2.5 m in the Z direction into two, as shown in FIG. 32(A) and (B). [Condition 1] Distance Z0 = 2.5 m, exposure time te = 0.8 ms, Figure 32 (A) [Condition 2] Distance Z0 = 1.5 m, exposure time te = 4.0 ms, Figure 32 (B) In this example, the conditions of the light irradiation unit 10 and the imaging unit 20 are such that the imaging area 80 of the light curtain 300 extends in the -Z direction. In this case, the object 400 is detected by the light curtain 300 under [Condition 1], but the object 400 is not detected by the light curtain 300 under [Condition 2].

[0095] Next, as shown in FIGS. 32(C) and (D), the following two light curtains 300 are used so as to divide the range of 1.5 to 2.5 m in the Z direction into two. [Condition 3] Distance Z0 = 2.5 m, exposure time te = 0.3 ms, Figure 32 (C) [Condition 4] Distance Z0 = 2.0 m, exposure time te = 0.5 ms, Figure 32 (D) In this example, the conditions of the light irradiation unit 10 and the imaging unit 20 are such that the imaging area 80 of the light curtain 300 extends in the -Z direction. In this case, the object 400 is not detected by the light curtain 300 under [Condition 3], but the object 400 is detected by the light curtain 300 under [Condition 4].

[0096] Figures 33(A) to (C) are diagrams showing an example (part 2) of the search operation in the case of Figure 29(A). Figures 33(A) and (B) show the process following Figures 32(C) and (D). As shown in Figures 33(A) and (B), the following two light curtains 300 are used to divide the range of 1.5 to 2.0 m in the Z direction into two. [Condition 5] Distance Z0 = 2.0 m, exposure time te = 0.15 ms, Figure 33(A) [Condition 6] Distance Z0 = 1.75 m, exposure time te = 0.25 ms, Figure 33(B) In this example, the conditions of the light irradiation unit 10 and the imaging unit 20 are such that the imaging area 80 of the light curtain 300 extends in the -Z direction. In this case, the object 400 is not detected by the light curtain 300 under [Condition 5], but the object 400 is detected by the light curtain 300 under [Condition 6]. By repeating the same process, it is possible to detect the range in which the object 400 exists (for example, within the trapezoid in FIG. 33(C)).

[0097] Also, by scanning the area where the object 400 exists with a small light curtain 300 in the Z direction with an exposure time te=0.1 ms, for example, the three-dimensional shape of the object 400 can be acquired quickly and with high accuracy. If the object 400 is, for example, a human, scanning an area of ​​50 cm at most is sufficient, and the remaining 1.5 m of scanning time is wasted. When trying to obtain a positional accuracy in the Z direction in mm units, scanning is required at intervals of about 10 mm, so there is a time of approximately 3 seconds spent scanning areas where there are no objects, and this time can be reduced.

[0098] 34(A) to (C) are diagrams showing another example of the search operation in the case of FIG. 29(A). When trying to divide the area in the Z direction into two, one of the boundaries in the ±Z direction of the imaging area 80 of the light curtain 300 is curved, so that it is not possible to divide it into two accurately. In order to prevent an object from being overlooked, it is desirable to overlap the imaging area 80 of the light curtain 300 covering 1.5 m to 2.5 m in the Z direction and the imaging area 80 of the light curtain 300 covering 0.5 m to 1.5 m in the Z direction at about 1.4 m to 1.5 m, as shown in FIG. 34(A) and (B). An object present in the overlapping area is detected by both light curtains 300, but since the overlapping area is known, the Z-direction position of the object can be easily identified.

[0099] 4. Third embodiment In the third embodiment, a case where the width of the laser light is taken into consideration in the cases of Figs. 1 to 9 will be described. Fig. 35 is a schematic diagram showing a configuration of a three-dimensional sensing device 3 according to the third embodiment. The three-dimensional sensing device 3 has a configuration similar to that of the three-dimensional sensing device 1 or 2 according to the first or second embodiment. The three-dimensional sensing device 3 according to the third embodiment is a device that can detect whether an object (a person in Fig. 35) is present in a box-shaped target area (a "U-shaped" target area) in a target space, and the shape of the object. The box-shaped target area is an area that is partitioned by, for example, surfaces 91, 92, 93 constituting a wall or a partition (a free-standing partition), and has one or more entrances 94, etc.

[0100] 36(A) and (B) are diagrams showing an example of a search operation of the three-dimensional sensing device 3. In the search operation, the three-dimensional sensing device 3 creates a light curtain 300 so as to surround the target area with planes 91, 92, 93, and 94, and can detect the intrusion of an object 400 (for example, a person) into the target area. However, if an object exists in front of the target area and there is an area that is shaded by the object, the accuracy of intrusion detection into the target area surrounded by the planes decreases. Therefore, a thick light curtain 300 is projected on the front side of the target area to determine the presence or absence of the object 400. To determine the intrusion detection, the position information of the object 400 in the Z direction is not required, and it is only necessary to know whether the object 400 is present on the front side of the target area. This makes it possible to appropriately monitor the entry and exit of people into and out of the target area.

[0101] Moreover, Fig. 37 is a diagram showing an example of the operation of the three-dimensional sensing device 3 according to embodiment 3. Fig. 38(A) is a diagram showing the angular function θ(t) of the irradiation direction of the light irradiation unit 10 and the angular function φ(t) of the line of sight direction of the image capture unit 20, Fig. 38(B) is a diagram showing the camera trigger in the image capture unit 20, and Fig. 38(C) is a diagram showing the on / off of the laser light of the light irradiation unit 10.

[0102] The three-dimensional sensing device 3 can form a light curtain 300 of any shape, not just a plane, by adjusting the synchronization timing between the irradiation direction of the light irradiation unit 10 and the line of sight direction of the imaging unit 20, although there are hardware restrictions. For example, as shown in FIG. 37, a light curtain 300 can be created that has two sides (91, 93) parallel to the Z axis and one side (92) perpendicular to the Z axis, and has a "U-shape" shape when viewed from above. The light curtain 300 can be constructed in a common area 75 surrounded by both the projection range (between 71 and 72) of the light irradiation unit 10 and the angle of view (between 73 and 74) of the imaging unit 20. For example, when the imaging unit 20 is used by rotating the light irradiation unit 10 by 90°, each row is scanned at a constant speed, and when the scanning direction is clockwise (from 73 to 74), the light curtain 300 is imaged in the order of points A1, A2, A3, and A4 in FIG. 37. The angular function φ(t) of the line of sight direction of the imaging unit 20 is expressed by the following equation (9) using a constant coefficient k determined by the frame rate and a minimum angle φmin of φ(t). φ(t)=k*t+φmin (9)

[0103] Since it is generally difficult to operate the imaging unit 20 at a line scanning speed other than a constant speed, the control unit 30 controls the angle function θ(t) of the irradiation direction of the light irradiation unit 10 to match the angle function φ(t) of the line of sight direction of the imaging unit 20. When the "U-shaped" light curtain 300 is made to have the largest size in the common area 75, point A1 is placed on the irradiation direction 71 of the minimum angle θmin of the light irradiation unit 10. If the distance of the surface 91 on the -X side of the "U-shaped" is X1 and the distance of the end point is Z1, the coordinates of point A1 are (X1, Z1). The angle function θ(t) of the light irradiation unit 10 for making the imaging area of ​​the light curtain 300 on the -X side side is expressed by the following formula (10).

[0104]

number

[0105] The irradiation direction of the light irradiation unit 10 is determined by the angle θ A2 and reaches point A2(X1,Z2).

[0106]

number

[0107] After reaching point A2, the angle function θ(t) is switched to the function expressed by the following equation (12) that creates a plane surface 92.

number

[0108] Next, the light irradiation unit 10 is rotated so that the angle θ expressed by the following formula (13) is A3 is reached.

[0109]

number

[0110] In order to image the area of ​​surface 93 on the +X side of the "U-shape," the light irradiation unit 10 and the imaging unit 20 are synchronized with the angle function θ(t) expressed by the following equation (14).

[0111]

number

[0112] When the imaging unit 20 reaches the line of sight 74 at the maximum angle φmax, the light curtain 300 consisting of a "U-shaped" imaging area is completed. Here, in order to make point A4 have the same distance Z1 as point A1, X2 is adjusted. With the "U-shaped" light curtain 300, it is possible to continue imaging at 50 Hz, for example, and detect an object crossing the side of the "U-shape".

[0113] FIG. 39 is a diagram showing an example of the operation of the three-dimensional sensing device 3 according to the third embodiment. FIG. 40(A) is a diagram showing the angle function θ(t) of the irradiation direction of the light irradiation unit 10 and the angle function φ(t) of the line of sight direction of the imaging unit 20, FIG. 40(B) is a diagram showing the camera trigger in the imaging unit 20, and FIG. 40(C) is a diagram showing the on / off of the light irradiation unit 10. For example, as shown in FIG. 39, a flat light curtain can be constructed at the entrance 94 so as to connect points A1 and A4 of the "U-shaped" light curtain 300, thereby making it possible to create a light curtain 300 that monitors the boundary of an area surrounded by a rectangular shape ("U-shaped") when viewed from above. As shown in FIGS. 40(A) to (C), it is preferable to control the angle function θ(t) of the light irradiation unit 10 by alternately setting it to the "U-shaped" and the flat type.

[0114] The "U-shaped" and "Rectangular" light curtains 300 can detect objects present on each side, and can therefore be used as sensors to detect an object 400 entering or leaving an enclosed area. In such a case, for example, there may be a case where it is not necessary to identify the position within the enclosed area, but it is only necessary to know whether the object 400 is present or not. The exposure time te is controlled to adjust the enclosed area to be the imaging area.

[0115] 41(A) to (D) are diagrams showing an example of a search operation of the three-dimensional sensing device 3. FIG. 42(A) and (B) are diagrams showing an example of a search operation of the three-dimensional sensing device 3. FIG. 41(A) to (D) show the results of calculating the imaging area with exposure times te of 0.1 ms, 0.5 ms, 1.0 ms, and 2.0 ms when a "U-shaped" light curtain 300 with a width of about 0.9 m is constructed in the range of Z=1.0 m to 2.0 m. Conditions were set so that the imaging area would expand inside the "U-shaped" light curtain 300, and the imaging area was constructed along each side of the "U-shaped". When the exposure time te is extended, the imaging area expands to fill the inside of the "U-shaped". With an exposure time of 2.0 ms, the inside of the "U-shaped" is completely filled with the imaging area. For example, as shown in Fig. 41 (A) to (D), an exposure time of 1.4 ms is the minimum exposure time for the imaging area to fill the inside of the "U-shape". At this time, since there is a portion of the imaging area that protrudes from the area surrounded by the "Rectangle", it is possible that an object outside the "Rectangle" will be detected. In this case, for example, a planar light curtain 300 is constructed at a position of Z = 1.0 m, and the exposure time is lengthened to 3.0 ms, so that the imaging area is the -Z side of 1.0 ms. An object detected in both Fig. 41 (A) and (B) can be determined to be outside the "Rectangle" area.

[0116] When monitoring an enclosed area such as a "U-shape" or "Rectangle-shape," control may be performed so that objects crossing each side are monitored normally with a short exposure time te of 0.1 ms, and the presence or absence of an object 400 inside is checked from time to time, for example, about once per minute. Alternatively, imaging with a short exposure time and a long exposure time may be alternately repeated. [Explanation of symbols]

[0117] 1 3D sensing device, 10 light irradiation section (illumination section), 20 imaging section (camera), 30 control section (control circuit), 100 light irradiation area, 200 line imaging range, 80 imaging area (detection area), 300 light curtain, 400 object, X second direction, Y first direction, Z depth direction.

Claims

1. A three-dimensional sensing device for detecting an object present in a target space, comprising: A light irradiation unit that forms a light irradiation area extending in a first direction by irradiating light and moves the light irradiation area in a second direction that is a direction intersecting the first direction; an imaging unit that performs an imaging operation to capture an image of a line imaging range extending in the first direction and moves the line imaging range in the second direction; a control unit that controls the light irradiation unit and the imaging unit so as to form a light curtain having a planar imaging area that is an intersection area where the light irradiation area and the line imaging area intersect, and detects the object overlapping the light curtain based on an image captured by the imaging unit; having The control unit is calculating a relationship between an exposure time of the imaging unit and a thickness of the light curtain in the depth direction for each position in the depth direction corresponding to a distance from the imaging unit; changing a position of the light curtain in the depth direction by controlling a light irradiation direction of the light irradiation unit and a line of sight direction of the imaging unit; changing a thickness of the light curtain in the depth direction by controlling an exposure time of the imaging unit for each position in the depth direction; A three-dimensional sensing device comprising:

2. A three-dimensional sensing device for detecting an object present in a target space, comprising: A light irradiation unit that forms a light irradiation area extending in a first direction by irradiating light and moves the light irradiation area in a second direction that is a direction intersecting the first direction; an imaging unit that performs an imaging operation to capture an image of a line imaging range extending in the first direction and moves the line imaging range in the second direction; a control unit that controls the light irradiation unit and the imaging unit so as to form a light curtain having a planar imaging area that is an intersection area where the light irradiation area and the line imaging area intersect, and detects the object overlapping the light curtain based on an image captured by the imaging unit; having The control unit is changing a depth direction position of the light curtain corresponding to a distance from the imaging unit by controlling a light irradiation direction of the light irradiation unit and a line of sight direction of the imaging unit; changing a thickness of the light curtain in the depth direction by controlling an exposure time of the imaging unit; performing a search operation to detect a first detection position, which is a position of the object in the depth direction, while changing a position of the light curtain in the depth direction by a first interval; A detection operation is performed to detect a second detection position, which is a position of the object in the depth direction, while changing a position of the light curtain in the depth direction by a second interval narrower than the first interval in a range around the first detection position. A three-dimensional sensing device comprising:

3. A three-dimensional sensing device for detecting an object present in a target space, comprising: A light irradiation unit that forms a light irradiation area extending in a first direction by irradiating light and moves the light irradiation area in a second direction that is a direction intersecting the first direction; an imaging unit that performs an imaging operation to capture an image of a line imaging range extending in the first direction and moves the line imaging range in the second direction; a control unit that controls the light irradiation unit and the imaging unit so as to form a light curtain having a planar imaging area that is an intersection area where the light irradiation area and the line imaging area intersect, and detects the object overlapping the light curtain based on an image captured by the imaging unit; having The control unit is changing a depth direction position of the light curtain corresponding to a distance from the imaging unit by controlling a light irradiation direction of the light irradiation unit and a line of sight direction of the imaging unit; changing a thickness of the light curtain in the depth direction by controlling an exposure time of the imaging unit; setting the thickness of the light curtain in the depth direction to a first thickness, and executing a search operation to detect a first detection position that is a position of the object in the depth direction; A detection operation is performed to change the thickness of the light curtain to a second thickness that is thinner than the first thickness in a range around the first detection position, and to detect a second detection position that is a position of the object in the depth direction. A three-dimensional sensing device comprising:

4. A three-dimensional sensing device for detecting an object present in a target space, comprising: A light irradiation unit that forms a light irradiation area extending in a first direction by irradiating light and moves the light irradiation area in a second direction that is a direction intersecting the first direction; an imaging unit that performs an imaging operation to capture an image of a line imaging range extending in the first direction and moves the line imaging range in the second direction; a control unit that controls the light irradiation unit and the imaging unit so as to form a light curtain having a planar imaging area that is an intersection area where the light irradiation area and the line imaging area intersect, and detects the object overlapping the light curtain based on an image captured by the imaging unit; having The control unit is changing a depth direction position of the light curtain corresponding to a distance from the imaging unit by controlling a light irradiation direction of the light irradiation unit and a line of sight direction of the imaging unit; changing a thickness of the light curtain in the depth direction by controlling an exposure time of the imaging unit; a searching operation is performed to detect a first detection position, which is a position of the object in the depth direction, while setting the thickness of the light curtain in the depth direction to a first thickness and changing the position of the light curtain in the depth direction by a first interval; A detection operation is performed to change the thickness of the light curtain to a second thickness that is thinner than the first thickness in a range around the first detection position, and detect a second detection position that is a position of the object in the depth direction while changing the position of the light curtain in the depth direction by a second interval that is narrower than the first interval. A three-dimensional sensing device comprising:

5. The control unit is The width of the light curtain in the second direction is set to a first width, and the searching operation is performed; and performing the detection operation by setting the width of the light curtain in the second direction to a second width narrower than the first width in a range around the first detection position.

5. The three-dimensional sensing device according to claim 2, wherein the three-dimensional sensing device is a three-dimensional sensor.

6. The position of the imaging unit is a position downstream of the position of the light irradiation unit in the moving direction of the light irradiation area.

5. The three-dimensional sensing device according to claim 1, wherein the three-dimensional sensing device is a three-dimensional sensor.

7. The position of the imaging unit is a position on the upstream side of the position of the light irradiation unit in the moving direction of the light irradiation area.

5. The three-dimensional sensing device according to claim 1, wherein the three-dimensional sensing device is a three-dimensional sensor.

8. The light irradiation unit and the image capturing unit are arranged side by side in the second direction.

5. The three-dimensional sensing device according to claim 1, wherein the three-dimensional sensing device is a three-dimensional sensor.

9. The imaging unit is a line sensor or a rolling shutter camera.

5. The three-dimensional sensing device according to claim 1, wherein the three-dimensional sensing device is a three-dimensional sensor.

10. The light irradiation unit includes a laser light source that emits the light, and a scanning mirror that scans the light in the first direction and the second direction.

5. The three-dimensional sensing device according to claim 1, wherein the three-dimensional sensing device is a three-dimensional sensor.

11. The control unit: The light curtain is formed in an arbitrary shape by adjusting the synchronization timing between the light irradiation direction of the light irradiation unit and the line of sight direction of the imaging unit, and the light curtain is adjusted to be a predetermined enclosed area by controlling the exposure time of the imaging unit.

5. The three-dimensional sensing device according to claim 1, wherein the three-dimensional sensing device is a three-dimensional sensor.

Images