Image sensing device

Abstract

An image sensing device (100) comprises a laser scanner (1), a camera (2), and a synchronization circuit (4). The laser scanner (1) is capable of causing a laser to scan in a direction parallel to an X-axis. The camera (2) is capable of causing an imaging region to scan in a direction parallel to the X-axis. The synchronization circuit (4) controls the laser scanner (1) and the camera (2) such that an emission direction of the laser and a line-of-sight direction of the camera (2) intersect one another on a two-dimensional plane within a three-dimensional space. The camera (2) images an object (41) intersecting the two-dimensional plane by scanning in the -X direction of the X-axis. The synchronization circuit (4) acquires an image of the object (41) by synchronizing the scanning of the laser scanner (1) and the camera (2) in the -X direction.

Description

Image Sensing Device

[0001] The present disclosure relates to image sensing devices.

[0002] In recent years, a sensing method has been proposed that uses an illumination plane created by a line laser and a sensing plane sensed by a line sensor to detect an object (see, for example, Patent Document 1).

[0003] Special Publication No. 2022-530349

[0004] In conventional technology, the area that can be sensed by a projection light curtain sensor is limited to the area where the light curtain surface is installed, and there is a problem in that it is not possible to capture the background or surroundings behind the sensing area.

[0005] An object of the present disclosure is to solve the above-mentioned problems and to generate an image that includes an image of a sensing area and an image behind or around the sensing area.

[0006] an image sensing device according to one aspect of the present disclosure, comprising: a laser scanner capable of scanning a laser extending in a direction parallel to a Y axis in an XYZ Cartesian coordinate system in a direction parallel to an X axis in the XYZ Cartesian coordinate system; a camera capable of scanning an imaging area extending in a direction parallel to the Y axis in a direction parallel to the X axis; and a synchronization circuit that controls the laser scanner and the camera, wherein the laser scanner and the camera are arranged along the X axis in the XYZ Cartesian coordinate system, the synchronization circuit controls the laser scanner and the camera so that an irradiation direction of the laser emitted from the laser scanner and a line of sight direction of the camera intersect on a two-dimensional plane in three-dimensional space, the camera images an object intersecting the two-dimensional plane by scanning in a −X direction of the X axis, the synchronization circuit synchronizes the scanning of the laser scanner and the camera in the −X direction to acquire an image of the object, and the synchronization circuit acquires an image of the background of the object by making the shutter open time of the camera longer while the laser is being irradiated than when the scanning in the −X direction is being performed, The synchronization circuit generates a composite image by combining the image of the object and the image of the background.An image sensing device according to another aspect of the present disclosure includes: a laser scanner capable of scanning a laser extending in a direction parallel to a Y axis in an XYZ Cartesian coordinate system in a direction parallel to an X axis in the XYZ Cartesian coordinate system; a camera capable of scanning an imaging area extending in a direction parallel to the Y axis in a direction parallel to the X axis; and a synchronization circuit that controls the laser scanner and the camera, wherein the laser scanner and the camera are arranged along the X axis in the XYZ Cartesian coordinate system, the synchronization circuit controls the laser scanner and the camera so that the irradiation direction of the laser emitted from the laser scanner and the line of sight direction of the camera intersect on a two-dimensional plane in three-dimensional space, the camera captures an image of an object intersecting with the two-dimensional plane by scanning in a −X direction of the X axis, the synchronization circuit synchronizes the scans of the laser scanner and the camera in the −X direction to obtain a first scanned image, and the synchronization circuit obtains a second scanned image captured by the camera in a state where the laser is not being irradiated. The synchronization circuit generates a difference image based on the difference between the first scanned image and the second scanned image.

[0007] According to the present disclosure, an image can be generated that includes an image of the sensing area and an image behind or around the sensing area.

[0008] 1 is a perspective view schematically illustrating the configuration of an image sensing device for performing light curtain sensing in the present disclosure. FIG. 2 is a diagram illustrating an example of the configuration of a laser scanner. (A) to (C) are diagrams respectively illustrating a 3D image of an object, an image of the object's background, and a composite image of these. FIG. 3 is a diagram illustrating the generation of a light curtain in the XZ plane. FIG. 4 is a diagram illustrating the operation of a light curtain on a virtual screen. FIG. 5 is a diagram illustrating the operation of a light curtain on a virtual screen. FIG. 6 is a diagram illustrating a control method for alternately acquiring sensing using a light curtain and a background image. FIG. 7 is a diagram illustrating a geometric arrangement for generating a light curtain. FIG. 8 is a diagram illustrating an example of the operation of an image sensing device for measuring an object. FIG. 9 is a diagram illustrating the line scanning operation of a camera. FIG. 10 is a diagram illustrating the area (hatched portion) in the XZ plane photographed in section P2(1) when the installation position (point Q) of the laser scanner is set as the origin. FIG. 11 is a graph showing the results of calculating the area in which a background image can be acquired when T remains at 16 ms and the exposure time te = 4 ms. 10 is a graph showing the results of calculating an area in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 6 ms. FIG. 11 is a graph showing the results of calculating an area in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 8 ms. FIG. 12 is a graph showing the results of calculating an area in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 32 ms. FIG. 13 is a graph showing the results of calculating an area in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 64 ms. FIG. 14 is a diagram showing an example of the operation of a laser scanner and a camera in embodiment 2. FIG. 15 is a graph showing the results of calculating an area in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 16 ms. FIG. 16 is a graph showing the results of calculating an area in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 32 ms. 10A and 10B are graphs showing the results of calculations of the area in which a background image can be acquired when T=16 ms and the exposure time te=64 ms, respectively, and when T=16 ms and the exposure time te=128 ms, respectively.23A to 23C are perspective views schematically showing the configuration of an image sensing device for performing light curtain sensing according to embodiment 3. Fig. 23A to Fig. 23C are views showing images generated in embodiment 3. Fig. 23B are views showing the operation of a laser scanner and a camera in embodiment 3. Fig. 23C are views showing the operation of a laser scanner and a camera in embodiment 4. Fig. 23D are views showing the operation of a laser scanner and a camera in embodiment 4.

[0009] <Embodiment 1> <Image sensing device 100> In the XYZ Cartesian coordinate system shown in each figure, the Z-axis direction (Z-axis) indicates a direction perpendicular to the virtual screen, the Y-direction (Y) indicates a direction perpendicular to the Z-axis direction and parallel to the imaging area, and the X-direction (X-axis) indicates a direction perpendicular to both the Z-axis direction and the Y-axis direction.

[0010] 1 is a perspective view showing a schematic configuration of an image sensing device 100 (also referred to as a light curtain sensor device) for performing light curtain sensing in the present disclosure. In the example shown in FIG. 1, a light curtain 25 is generated by a combination of a laser scanner 1 and a camera 2.

[0011] The image sensing device 100 includes a laser scanner 1, a camera 2, and a synchronization circuit 4. In the image sensing device 100, the laser scanner 1 and the camera 2 are arranged in parallel along the X direction, and are controlled by the synchronization circuit 4.

[0012] A laser 12 (also called a line laser) that spreads in the Y direction is emitted from the laser scanner 1. A virtual screen 10, which is a plane for the purpose of explanation, exists perpendicular to the Z-axis direction at a position Z0 away from the laser scanner 1 in the Z direction. The laser emitted from the laser scanner 1 appears as a linear laser 12 that extends in the Y direction on the virtual screen 10.

[0013] <Laser Scanner 1> Figure 2 is a diagram showing an example of the configuration of a laser scanner. The laser scanner 1 emits a laser. The laser scanner 1 can scan in a direction parallel to the X axis with a laser extending in a direction parallel to the Y axis of an XYZ Cartesian coordinate system. The laser scanner 1 and camera 2 are arranged along the X axis of the XYZ Cartesian coordinate system.

[0014] A laser beam with wavelength λ is emitted from a laser light source 5, and is converted into a laser beam 12 that spreads in the Y direction by a line beam generating element 6. The line beam generating element 6 is, for example, a cylindrical lens or a Powell lens. The laser beam 12 is deflected in the Z direction by a galvanometer mirror 7. The galvanometer mirror 7 can be swung around the Y axis at angles of ±α, and the laser beam 12 is scanned around the Y axis within an angle range of ±2α. On the virtual screen 10, the laser beam 12 is scanned from the +X direction to the −X direction, and the entire area of ​​the scan range 13 in FIG. 1 or 2 is irradiated.

[0015] <Camera 2> Camera 2 can scan an imaging area 22 (also called a linear imaging area) extending in a direction parallel to the Y axis of an XYZ Cartesian coordinate system in a direction parallel to the X axis of the XYZ Cartesian coordinate system. For example, camera 2 captures an image of an object (also called a target object) that intersects with a two-dimensional plane in three-dimensional space by scanning in the −X direction of the X axis.

[0016] In this embodiment, camera 2 is a rolling shutter camera. Camera 2 reads out luminance information of image sensor pixels sequentially, starting from the top row. By shortening the exposure time of camera 2 under the control of synchronization circuit 4, a linear imaging area extending in the row direction of the image sensor can be scanned in the column direction perpendicular to the row direction. For example, in a 1000-row rolling shutter camera operating at a frame rate of 50 fps, if the exposure time per row is set to te = 20 μs, adjacent rows can be separated in time and sensing can be performed row by row. In actual operation, synchronization circuit 4 controls camera 2 to shift the exposure area in sequence by five rows, for example, with an exposure time of te = 100 μs.

[0017] Normally, a rolling shutter camera is used so that the row direction is horizontal, but in this embodiment, camera 2 is installed rotated -90° around its optical axis. Camera 2 is installed so that the optical axis direction of camera 2 faces slightly inward from the Z axis direction (-X direction) so that the entire imaging range 23 of camera 2 on virtual screen 10 roughly overlaps with the entire laser scanning range 13. However, even if camera 2 faces in the Z axis direction, if there is overlap between the entire imaging range 23 and the entire scanning range 13, a light curtain 25 is generated within the overlapping region.

[0018] The camera 2 is also equipped with a bandpass filter 8. The bandpass filter 8 may be installed at the tip of the lens barrel of the camera 2 or inside the camera 2. The bandpass filter 8 is a wavelength filter that passes only light of wavelength λ emitted from the laser light source 5.

[0019] <Synchronization Circuit 4> The synchronization circuit 4 controls the laser scanner 1 and the camera 2. The synchronization circuit 4 can synchronize the operation of the laser scanner 1 and the operation of the camera 2. For example, the synchronization circuit 4 controls the laser scanner 1 and the camera 2 so that the irradiation direction of the laser emitted from the laser scanner 1 and the line of sight direction of the camera 2 intersect on a two-dimensional plane in three-dimensional space.

[0020] FIGS. 3A to 3C are diagrams showing a 3D image of an object, an image of the object's background, and a composite image of these, respectively. In this embodiment, the image sensing device 100 can generate a composite image by acquiring a 3D image of the object and an image of the object's background and combining these images. For example, as shown in FIG. 3A, the synchronization circuit 4 acquires an image of the object by synchronizing the scans (also referred to as the first scan) of the laser scanner 1 and camera 2 in the −X direction. As shown in FIG. 3B, the synchronization circuit 4 acquires an image of the object's background by lengthening the shutter open time (also referred to as the exposure time) of the camera 2 while the laser is irradiated compared to the first scan in the −X direction. The image of the object's background is also referred to as a background image. The background image is acquired, for example, by scanning the laser scanner 1 and camera 2 from the +X direction to the −X direction of the X axis.

[0021] The synchronization circuit 4 generates a composite image by combining the image of the object and the background image. As shown in Fig. 3C, this process enables the image sensing device 100 to generate a composite image in which a 3D image of the object and a background image are combined. That is, the image sensing device 100 can generate an image that includes an image of the sensing area and an image behind or around the sensing area.

[0022] <Light Curtain Sensing> Fig. 4 is a diagram illustrating the generation of a light curtain 25 in the XZ plane. Figs. 5 and 6 are diagrams illustrating the operation of the light curtain 25 on the virtual screen 10. In Fig. 4, the angle θ formed by the irradiation direction 11 of the laser 12 from the Z-axis direction and the Z-axis is taken as θ. Note that the clockwise direction is positive. In Fig. 4, the angle θa formed by the irradiation direction 11a and the Z-axis at a certain time ta has a negative value. Similarly, the angle θb formed by the irradiation direction 11b and the Z-axis at time tb has a positive value. Note also that the positive direction of the X-axis is to the left in the diagram.

[0023] In FIG. 4, the camera 2 is installed rotated -90° around the optical axis, so the line of sight 21 of the camera 2 scans clockwise within the XZ plane. The angle with the Z axis is designated as φ. In FIG. 4, the angle φa between the line of sight 21a and the Z axis at time ta and the angle φb between the line of sight 21b and the Z axis at time tb are both positive values. Here, at time ta, the irradiation direction 11a and the line of sight 21a intersect at an intersection 24a on the virtual screen 10a. Similarly, at time tb, the irradiation direction 11b and the line of sight 21b intersect at an intersection 24b on the virtual screen 10a. FIGS. 5 and 6 show the relationship between the laser 12 and the imaging area 22 on the virtual screen 10a at a distance Za and on the virtual screen 10b at a distance Zb.

[0024] On the virtual screen 10a, scanning is performed so that the laser 12 and the imaging area 22 always overlap. As will be explained later, such an operation is possible by appropriately controlling the angle of the galvanometer mirror 7 with respect to time. At this time, since the virtual screen 10b is installed at a different distance from the virtual screen 10a, as is clear from FIG. 6, the laser 12 and the imaging area 22 never overlap.

[0025] With the light curtain 25 generated on the virtual screen 10a, when the light curtain 25 intersects with an object (also referred to as the target object 41), an image outside the intersecting area is not captured by the camera 2. This is because the exposure time for each pixel in one frame is extremely short (for example, exposure time te = 60 μs), and the bandpass filter 8 cuts off light rays other than those with the laser wavelength before they reach the image sensor inside the camera 2. Therefore, the background is not captured by light curtain sensing.

[0026] Here, the conditions for scanning the virtual screen 10a so that the laser 12 and the imaging area 22 overlap will be described. First, when the laser 12 is linear and extends in the Y direction, the imaging area 22 of the camera 2 must extend precisely in the Y direction. To achieve this, a fine adjustment mechanism is provided for finely adjusting the orientation of the camera 2 around its optical axis. Furthermore, the imaging lens of the camera 2 must be a distortion-corrected lens. Otherwise, the imaging area 22 will be distorted from its linear shape. Furthermore, to scan the virtual screen 10a while keeping the laser 12 and the imaging area 22 overlapping, the imaging start timing of the camera 2 and the scanning start timing of the laser scanner 1 must be synchronized with an appropriate delay, and a synchronization circuit 4 is used for this purpose. Furthermore, the scanning speed of the laser 12 and the scanning speed of the imaging area 22 must be the same on the virtual screen 10a.

[0027] 7 is a diagram illustrating a control method for alternately capturing sensing using a light curtain and background images. In FIG. 7, the dashed line represents the angular function φ(t) of the line of sight direction of the camera 2 in the imaging area, and the solid line represents the angular function θ(t) of the irradiation direction 11 of the laser 12. In sections P1 and P3, a planar light curtain perpendicular to the Z axis is generated at a position at a different distance Z0, and background images are captured in sections P2 and P4. The specific parameters used to calculate the graph shown in FIG. 7 will be described later.

[0028] First, the operation in section P1 will be described. Figure 8 is a diagram showing the geometric arrangement for generating the light curtain 25. The irradiation direction 11 of the laser 12 and the line of sight direction 21 of the camera 2 intersect on a virtual screen 10a placed at a distance Z0 in the Z-axis direction. As shown in Figure 8, the angular function θ(t) of the irradiation direction 11 of the laser 12 and the angular function φ(t) of the line of sight direction 21 of the camera 2 are angles from the Z-axis direction. In the arrangement shown in Figure 8, both the angle θ and the angle φ have positive values. The angle of the galvanometer mirror is half this angle (α = θ(t) / 2). The angular function φ(t) of the line of sight direction of the camera 2 is expressed by the following equation (1) using a constant coefficient k determined by the frame rate: φ(t) = k × t + φmin (1)

[0029] Since it is generally difficult to operate the line scanning speed of a rolling shutter camera at anything other than a constant speed, the angular function θ(t) of the irradiation direction 11 of the laser 12 is controlled to match the angular function φ(t) of the line of sight direction of the camera 2. Using Figure 8, the relationship between θ(t) and φ(t) is found. The rotation axis of the laser scanner 1 is set to point Q, the rotation axis of the line of sight direction 21 of the camera 2 is set to point R, and the distance between them is set to b. The foot of the perpendicular line drawn from the intersection point 24 to the line QR is set to point P. Then, the signed distance PR can be expressed as Z0 * tan φ using φ. Furthermore, since the signed distance PQ can be expressed as Z0 * tan θ, the following equation (2) holds. Z0 × tan φ = Z0 × tan θ + b (2)

[0030] That is, when the angle φ is fixed, the angle θ of the laser 12 is calculated by the following equation (3): θ=Arctan(tan φ−b / Z0) (3)

[0031] When the virtual screen 10a is a plane perpendicular to the Z axis, the distance Z0 is a constant value. The function that generates the planar light curtain 25 at the distance Z0 is shown in the graph of section P1 in Figure 7. The specific parameters for plotting the angle function for section P1 are as follows:

[0032] <<Parameters for Section P1>> Scanning range of laser 12: -25°≦θ≦25° (θmin=-25°, θmax=25°) Scanning range of camera 2: -15°≦φ≦35° (φmin=-15°, φmax=35°) b=0.15m Time for camera 2 to scan the entire line T=16ms Z0=0.5m

[0033] The coefficient k in the formula (1) is expressed by the following formula (4): k=(φmax−φmin) / T (4)

[0034] Here, the camera 2 has a field angle of 50° in the ZX plane, and its optical axis is tilted by +10° with respect to the Z axis.

[0035] The position of the plane with parameters Z0 = 0.5 m in section P1 corresponds to the position marked on the virtual screen 10b in Figure 4. In the graph of section P1 shown in Figure 7, the angular function φ(t) of the line of sight direction of camera 2 varies linearly with time t, as expressed by equation (1), and varies within the range of φmin = -15° to φmax = 35° between times t = 0 and 16 ms. The angle θ of laser 12 during this period is given by equation (3), which is shown in the diagram below as the angular function θ(t) of the irradiation direction 11 of laser 12 in section P1(1). θ and φ each have limited scanning ranges, and a light curtain is generated only when both θ and φ are within their scanning ranges. In other words, the light curtain generation region is between points 26 and 27 in Figure 4.

[0036] The intersection of the irradiation direction of the laser 12 at the minimum angle θmin and the virtual screen 10b is designated as 26, and the intersection of the line of sight 21 of the camera 2 at the maximum angle φmax and the virtual screen 10b is designated as 27.

[0037] When the laser scan in section P1(1) is completed, the angle of the galvanometer mirror 7 must be returned in the negative direction in preparation for the next laser scan. Because the galvanometer mirror 7 is an object with a moment of inertia, a non-small finite time Tback is required to return it to the start position of the next scan. While this differs depending on the product specifications of the galvanometer mirror 7, for example, Tback = 4 ms is required. In section P1(2), the angle of the laser 12 is swung from +25° to -25° using a sine function.

[0038] <Section P3> In section P3, the light curtain 25 is set at a position with a distance Z0 = 2 m, which is different from that in section P1. The parameters other than Z0 are the same as those in section P1.

[0039] <<Operation of Synchronization Circuit>> A synchronization circuit 4 is used to operate the angular function φ(t) of the line of sight direction of the camera 2 and the angular function θ(t) of the irradiation direction 11 of the laser 12 at the precise timing shown in FIG. 7. The synchronization circuit 4 is pre-programmed with a function waveform that gives the angular function θ(t) of the irradiation direction 11 of the laser 12, and the timing for outputting a trigger pulse shown in FIG. 7 in accordance with the angular function θ(t). An analog voltage waveform that gives the angular function θ(t) is output from the synchronization circuit 4 to the laser scanner 1. A trigger pulse is output to the camera 2. The camera 2 starts capturing one frame of image at the rising edge of the trigger pulse voltage. In other words, it starts scanning the line capture area.

[0040] <Three-dimensional measurement using light curtain sensors> In Figure 7, planar light curtains 25 were generated at positions Z0 = 0.5 m and 2 m in sections P1 and P3, respectively. However, three-dimensional measurement can be performed by narrowing the spacing between the light curtains in the Z direction and installing multiple light curtains.

[0041] 9 is a diagram showing an example of the operation of an image sensing device that measures an object 41. In FIG. 9, a light curtain is installed in a section Pn at a distance Zn calculated by the following equations (5) and (6): n=1, 2, ..., N (N is a natural number) (5) Zn=Z0+(n-1)ΔZ (6) (n is a natural number)

[0042] That is, the distance between adjacent light curtains is ΔZ. An object 41 is placed within the sensing area of ​​such a light curtain. Images of cross sections of the object 41 where the light curtains 25a, 25b, 25c, ... intersect are captured by the camera 2. The cross-sectional area of ​​the object 41 reflected by the light curtain at a distance Zn is at a distance Zn from the light curtain sensor. Since many cross-sectional images at different distances Zn are acquired, distance information of the surface of the object 41, i.e., a depth map, can be obtained.

[0043] <<Background Image Acquisition Operation in Section P2>> An example of the operation of the camera 2 and laser scanner 1 for acquiring a background image is shown in sections P2 and P4 of FIG. 7. Note that the same operation is performed in sections P2 and P4. FIGS. 10(A) and 10(B) are diagrams showing the line scanning operation of the camera 2. FIG. 10(A) is a diagram showing the line scanning operation of the camera 2 when generating a light curtain. FIG. 10(B) is a diagram showing the line scanning operation of the camera 2 when acquiring a background image in section P2. In FIGS. 10(A) and 10(B), the horizontal direction represents the passage of time, and the vertical direction represents the row number of the rolling shutter camera 2. The horizontal width of the rectangular hatched area represents the time it takes to read an image in one row of pixels, i.e., the shutter open time te.

[0044] In the rolling shutter camera 2, pixels are read out sequentially, row by row, starting from the top row, resulting in a readout delay time tc for each row. For example, assuming T = 16 ms and 1,000 rows, tc = 16 μs. In FIG. 10(A), te is, for example, 80 μs. Generating a light curtain corresponds to the laser 12 irradiating each hatched area over time on the diagram in FIG. 10(A) when a planar object is present as an actual object on the light curtain. To function effectively as a light curtain sensor, it is desirable for the shutter open time te to be in a range from approximately the same as the transition time tc for each row of the camera 2 to approximately 20 times that. In the above specific example, the shutter open time te is in a range from 16 μs to 320 μs.

[0045] In FIG. 10(B), te is set to a very long value of, for example, 16 ms. After scanning a row of pixels (T = tc × 1000 = 16 ms), it takes te = 16 ms to capture the last row, so it takes (T + te) = 32 ms to capture one frame. The hatched parallelogram area in section P2 of FIG. 7 represents the shutter release of camera 2. As described above, camera 2 is rotated -90°, so the capture area of ​​the first row in FIG. 10(B) corresponds to the base of the parallelogram in the hatched area of ​​FIG. 7 (i.e., the line of sight angle of -15°). Because tc is the same in sections P1 and P2, the angular function φ(t) of the line of sight of camera 2 in section P2 is a straight line with the same slope k as in section P1. That is, the angular function φ(t) of the line of sight direction of camera 2 in section P2 is a straight line obtained by shifting equation (1) in the time direction on the horizontal axis.

[0046] The angular function θ(t) of the irradiation direction 11 of the laser 12 is a linear function of time, ranging from θmin = -25° to θmax = +25° at the time (T + te) when the image capture by the camera 2 ends. If the start time of the section P2 is t = t2a, then θ(t) for the section P2 is expressed by the following equations (7) and (8): θ(t) = kP2 × (t - t2a) + θmin (7) kP2 = (θmax - θmin) / (T + te) (8)

[0047] In other words, scanning is performed slowly at twice the speed of θ(t) in section P1. As shown in Figure 7, the laser is turned on in this section P2(1). When section P2(1) ends, the laser irradiation direction is returned to the negative direction in section P2(2) in preparation for generating a light curtain in the next section P3.

[0048] FIG. 11 is a diagram showing area 2 (hatched area) in the XZ plane captured in section P2(1) when the installation position (point Q) of laser scanner 1 is the origin. The dashed lines indicate the direction from point Q to the Z axis, where θmin = -25° and θmax = 25°. In other words, the entire scanning range of the laser is the area (referred to as area A) sandwiched between the two dashed lines. Similarly, the dotted lines indicate the direction from point R to the Z axis, where φmin = -15° and φmax = 35°. In other words, the entire capturing range of camera 2 is the area (referred to as area B) sandwiched between the two dotted lines.

[0049] The range in which a light curtain can be generated (referred to as area C) is the area where area A and area B overlap. In other words, area C is the area where area A and area B overlap. Area C can be classified into areas 1, 2, and 3 in FIG. 11. Area 2 is the range that is photographed in section P2(1) in FIG. 7, and only the very narrow areas of area 1 and area 3 are not photographed in section P2(1). Note that while FIG. 11 only illustrates the range Z≦5 m, area 2 extends into the area where Z>5 m by extending this range.

[0050] FIG. 12 is a graph showing the results of calculating the area 2 in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 4 ms. FIG. 13 is a graph showing the results of calculating the area 2 in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 6 ms. FIG. 14 is a graph showing the results of calculating the area 2 in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 8 ms. FIG. 15 is a graph showing the results of calculating the area 2 in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 32 ms. FIG. 16 is a graph showing the results of calculating the area 2 in which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 64 ms.

[0051] As shown in Figure 12, when te = 4 ms, region 2 is a band extending diagonally upward to the left, and background images from a sufficiently distant location cannot be captured. As shown in Figure 13, when te = 6 ms, region 2 extends far enough in the range of X > -0.2 m, and background images can be captured over a wide range. As shown in Figure 14, when te = 8 ms, region 2 expands even further. As shown in Figures 15 and 16, region 2 expands as Te = 16 ms, 32 ms, and 64 ms, but the increase in region 2 decreases as the exposure time increases.

[0052] Increasing the exposure time has the disadvantage of reducing the time available for light curtain sensing, so in practice, an exposure time of approximately te to 2T or less will be sufficient. In summary, even with an image sensing device 100 that generates a light curtain, a two-dimensional image of the entire background can be obtained by increasing the exposure time te. A specific guideline for the exposure time te is that it should be at least one-fourth the scan time TLC when generating the light curtain. In other words, the relationship between the exposure time te and the scan time TLC should preferably satisfy te > (1 / 4) × TLC.

[0053] Note that the exposure time te when generating the light curtain is shorter than the time T for the camera 2 to scan all lines (also referred to as the total scan time T), so the scan time TLC = (T+te) when generating the light curtain is approximately equal to the total scan time T of the camera 2. In this case, for example, the exposure time te (i.e., the shutter open time) is equal to or less than one-fourth of the total scan time T of the camera 2.

[0054] Combining this two-dimensional image of the entire background with a 3D image generated by a light curtain can clearly indicate the location of an object within the entire background, resulting in a visually easy-to-understand sensor. For example, as described above, a three-dimensional image 51 as shown in FIG. 3A can be obtained by generating multiple light curtains. The three-dimensional image 51 is, for example, a depth map in which the color changes depending on the distance. After obtaining the three-dimensional image, a background image 52 as shown in FIG. 3B is obtained. The background image 52 includes a two-dimensional image 53 of the object detected by the light curtain and a background image not detected by the light curtain. By combining the background image 52 and the three-dimensional image 51, a composite image 55 is obtained in which the 3D image is displayed as a depth map within a background image 54, as shown in FIG. 3C.

[0055] <<Use as a Safety Sensor>> In addition to being used as a sensor for acquiring 3D images, the light curtain sensor can also be used as a safety sensor. In this case, the sensing surface is set using one or several flat light curtains, rather than multiple light curtains as expressed in equation (6). Alternatively, the light curtain can easily be curved rather than flat. When used as a safety sensor, it can not only capture an image of an object that intersects with the light curtain, but also generate an image that combines a background image as shown in Figure 3(C). This makes it possible to create a safety sensor that is visually easy to understand.

[0056] <<Frequency of Background Image Acquisition>> In Figure 7, the operation of acquiring a background image after generating one light curtain is repeated, but the timing and frequency of background image acquisition can be changed depending on how the light curtain sensor is used. For example, if the image sensing device 100 is fixed and used as an intruder detection sensor, it is sufficient to acquire a background image when there is no intruder, and there is no need to frequently update the background image when performing light curtain sensing. On the other hand, if the light curtain sensor is installed on a moving vehicle and the background scene is constantly changing, it will be necessary to incorporate background image acquisition timing into the light curtain sensing and frequently update the background image.

[0057] 17 shows an example of the operation of the laser scanner 1 and the camera 2 in the second embodiment. In the present embodiment, the background image is acquired by scanning the camera 2 in the −X direction of the X axis while the irradiation direction of the laser scanner 1 is moving in the +X direction of the X axis.

[0058] 17 is a diagram showing an example of the operation of the image sensing device 100 in embodiment 2. In embodiment 2, the direction in which the laser 12 scans in section P2 differs from that in embodiment 1. In section P1 in Fig. 17, a light curtain is generated by the same operation as in section P1 shown in Fig. 7.

[0059] The angular function θ(t) of the irradiation direction 11 of the laser 12 in the section P2 is set to be expressed by the following equations (9) and (10), where t=t2a is the start time of the section P2. θ(t)=k′P2×(t−t2a)+θmax (9) k′P2=−(θmax−θmin) / (T+te) (10)

[0060] Because it is difficult to change the scanning direction of camera 2 due to the installation attitude of camera 2 (the angle of rotation around the optical axis) and the characteristics of the device, the angular function φ(t) of the line of sight direction of camera 2 can only be changed to the same direction as in section P1. In Figure 17, the angular function φ(t) of the line of sight direction of camera 2 in section P2 is a straight line with the same slope as in section P1, and the coefficient k of this slope is expressed by equation (4).

[0061] Fig. 18 is a graph showing the results of calculation of area 2 from which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 16 ms. Fig. 19 is a graph showing the results of calculation of area 2 from which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 32 ms. Fig. 20 is a graph showing the results of calculation of area 2 from which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 64 ms. Fig. 21 is a graph showing the results of calculation of area 2 from which a background image can be acquired when T remains at 16 ms and the exposure time te is set to 128 ms.

[0062] 18 to 21, region 2 enclosed by solid lines is the region imaged in section P2 in Fig. 17. When exposure time te = 16 ms, region 2 in this second embodiment is narrower, narrower than region 2 shown in Fig. 11 for the same exposure time. This is because, in the first embodiment, irradiation direction 11 of laser 12 and the line of sight direction of camera 2 are generally in the same direction, whereas in the second embodiment, irradiation direction 11 of laser 12 and the line of sight direction of camera 2 are scanned in opposite directions, so that no region can be imaged at the beginning of section P2.

[0063] When the exposure time te is 32 or 64 ms, region 2 expands rapidly. When the exposure time te is 128 ms, region 2 expands more than when te is 64 ms, but the increase is small. In this way, it is possible to acquire a background image even if the scanning direction of laser 12 is reversed. Image processing after acquiring the background image is the same as in embodiment 1.

[0064] Advantages of the Second Embodiment The difference from the first embodiment in generating a light curtain is that the angle θ of the laser 12 repeats monotonically increasing and decreasing for each section. That is, the light curtain is generated when the galvanometer mirror 7 rotates clockwise as viewed from above (forward rotation), and the background image is acquired when the galvanometer mirror 7 rotates counterclockwise (return rotation).

[0065] Therefore, the difference in angle when switching from section P1(1) to the next section P2(1) is smaller than in the first embodiment. This makes it possible to reduce the time Tback required to rotate the galvanometer mirror 7 to the start angle of the next section. For example, while Tback = 4 ms was required in the first embodiment, it can be reduced to Tback = 0.5 ms in the second embodiment. This makes it possible to reduce wasted time that is not related to generating the light curtain or acquiring the background image.

[0066] <Embodiment 3> Fig. 22 is a perspective view showing a schematic configuration of an image sensing device 100 for performing light curtain sensing according to embodiment 3. In the image sensing device of Fig. 22, the arrangement of the laser 1 and the camera 2 is reversed from left to right compared to the image sensing device 100 in Fig. 1, but this is not an essential problem according to embodiment 3. Figs. 23(A) to 23(C) are diagrams showing images generated in embodiment 3.

[0067] In the third embodiment, it is assumed that a high-brightness object 43 is included in the image capture scene, as shown in Fig. 22. This high-brightness object 43 is, for example, a light source such as a lamp, or a background object illuminated by sunlight during the day. As described in the first embodiment, the light curtain sensor is designed to block background light due to its configuration, but if the brightness of the high-brightness object 43 is high, its image will be captured by the camera 2.

[0068] As shown in FIG. 23A, a laser-on image 50 includes both an image 51 to be sensed by the light curtain and a high-brightness object image 58 formed by a high-brightness object 43 .

[0069] In this embodiment, the synchronization circuit 4 acquires a first scan image (hereinafter referred to as a laser-on image 50) by synchronizing the scans in the -X direction of the laser scanner 1 and the camera 2. Furthermore, the synchronization circuit 4 acquires a second scan image (hereinafter referred to as a laser-off image 56) captured by the camera 2 when the laser 12 is not being irradiated. The synchronization circuit 4 generates a difference image 57 that is generated by subtracting the laser-on image 50 from the laser-off image 56. This allows the image sensing device 100 to generate an image that includes only the image 51 sensed by the light curtain, without including the high-brightness object image 58.

[0070] Figure 24 is a diagram showing the operation of the laser scanner 1 and camera 2 in embodiment 3. Section P1 in Figure 24 represents a function for generating a planar light curtain 25 at a position Z0 = 0.5 m, similar to section P1 in Figure 7 . However, because the laser 1 and camera 2 are arranged left-right reversely, the function curve of the angle function θ(t) of the irradiation direction 11 of the laser 12 is different. The parameters for section P1 in Figure 24 are as follows: <Parameters for Section P1 in Figure 24> Scanning range of laser 12: -25° ≦ θ ≦ 25° (θmin = -25°, θmax = 25°) Scanning range of camera 2: -35° ≦ φ ≦ 15° (φmin = -35°, φmax = 15°) b = 0.15 m Time T for camera 2 to scan the entire line = 16 ms Z0 = 0.5 m In section P1 in Figure 24 , a light curtain is generated using the same operation as section P1 in Figure 7 . When a high-brightness object 43 is included in the measurement area, the laser-on image 50 acquired in section P1 includes both an image 51 sensed by the light curtain and an image 58 of the high-brightness object.

[0071] In section P2 in Figure 24, the galvanometer mirror 7 that scans the laser 12 and the camera 2 operate in exactly the same way as in section P1, except that the laser 12 is turned off. The image acquired at this time is shown in Figure 23(B). In this case, the sensing display image (laser-off image 56) does not show the target object 41, but only shows a high-brightness object image 58.

[0072] Fig. 23(C) shows a difference image 57 generated by subtracting the laser-off image 56 shown in Fig. 23(B) from the laser-on image 50 shown in Fig. 23(A). Fig. 23(C) does not show a high-brightness object image 58, and only shows an image 51 of the target object 41 sensed by the light curtain.

[0073] When the background scene changes from moment to moment, such as when the image sensing device 100 is mounted on a moving vehicle, or when the high-brightness object 43 moves even when the image sensing device 100 is fixed, the position and brightness of the high-brightness object 43 change within the field of view of the camera 2, so in such cases it is necessary to acquire the difference image 57 in real time. In such cases, as shown in sections P3 and P4 following sections P1 and P2 in Fig. 24, the laser can be repeatedly turned on and off for each frame to constantly acquire the laser-on image 50 and the laser-off image 56. Note that section P3 is a function for generating a planar light curtain at the position Z0 = 0.9 m.

[0074] If the image sensing device 100 is fixed and the high-brightness object 43 does not move, the laser-off image 56 does not need to be acquired frequently; it can be acquired at long time intervals (e.g., once an hour) when changes are expected. This allows the time (interval) used for acquiring background images to be used for light curtain sensing, enabling sensing at a faster frame rate. As shown in FIG. 24 , the galvanometer mirror 7 and camera 2 continue to operate in the same way as when the laser is on while acquiring the laser-off image 56. Therefore, simply by switching the laser on and off, the laser-on image 50 and the laser-off image 56 can be switched. In other words, the difference image 57 can be easily obtained without changing the program controlling the angular function θ(t) of the irradiation direction 11 of the laser 12.

[0075] <Fourth Embodiment> Fig. 25 shows a diagram illustrating the operations of the laser scanner 1 and the camera 2 in this fourth embodiment. As in the third embodiment, a scene is assumed in which there is a high-luminance object 43 as shown in Fig. 12.

[0076] In this embodiment, the laser-off image 56 (second scan image) is acquired by scanning the camera 2 in the −X direction of the X axis (positive angle direction) while the irradiation direction of the laser scanner 1 is moving in the +X direction of the X axis (negative angle direction).

[0077] 25 is a diagram showing the operation of the laser scanner 1 and the camera 2 in embodiment 4. Embodiment 4 differs from embodiment 3 in that the angular function θ(t) of the irradiation direction 11 of the laser 12 in sections P2 and P4 is monotonically decreasing, and the scanning direction of the laser in sections P2 and P4 is opposite to that in sections P1 and P3.

[0078] In section P1(1) of Fig. 25, a light curtain is generated that is exactly the same as section P1(1) of Fig. 24. After section P1(1) ends, the scanning of camera 2 has not yet finished, that is, the angular function φ(t) of the line of sight direction of camera 2 has not yet reached φmax = 15°, so the section until this finishes is designated as P1(2).

[0079] In section P2, the laser 12 is turned off as shown in Figure 25, so a laser-off image 56 corresponding to Figure 23(B) is acquired. Immediately after the camera starts scanning in section P1, section P2 begins with the start of the camera scanning. Although the laser is turned off, the galvanometer mirror 7 moves in the opposite direction to section P1, up to the starting angle of section P3, which generates the next light curtain. In Figure 25, the angular function θ(t) of the irradiation direction 11 of the laser 12 connects the angle at the end of section P1(1) and the start angle of section P3(1) with a gentle sine curve.

[0080] In section P3, a light curtain with a distance Z0 different from that in section P1 is generated, and in section P4, the laser is turned off and the galvanometer mirror 7 slowly rotates in the opposite direction to prepare for the generation of the next light curtain in section P4.

[0081] As described in the third embodiment, a difference image 57 showing only the light curtain image 51 can be obtained by taking the difference between the acquired laser-on image 50 and laser-off image 56 .

[0082] Effect of Embodiment 4 The difference between Embodiment 3 and Embodiment 4 is that a light curtain is generated when the galvanometer mirror 7 rotates clockwise as viewed from above (forward rotation), and a background image is acquired when it rotates counterclockwise (return rotation). In Figure 24, between the end of light curtain generation interval P1(1) and the start of the next light curtain generation interval P3(1), the galvanometer mirror 7 makes two return rotations and one forward rotation.

[0083] On the other hand, in Figure 25, there is only one backward rotation in the same section. Therefore, the time Tback required to rotate the galvanometer mirror 7 to the start angle of the next section can be reduced. In other words, there is an effect of reducing wasted time that is not related to generating a light curtain or acquiring a laser-off image. Furthermore, in sections P1(2), P2(2), etc. in Figure 24, the galvanometer mirror 7 is rotated at high speeds at its limit, which can cause deterioration of the galvanometer mirror 7. However, in Figure 25, there is no such high-speed operation, which has the advantage of reducing the factors that can cause failure of the galvanometer mirror 7.

[0084] The features of the above-described embodiments can be combined with each other.

[0085] REFERENCE SIGNS LIST 1 Laser scanner, 2 Camera, 4 Synchronization circuit, 6 Line beam generating element, 7 Galvanometer mirror, 10, 10a Virtual screen, 11, 11a, 11b Irradiation direction, 12 Laser, 21 Line of sight direction, 22 Imaging area, 25 Light curtain, 41 Object, 100 Image sensing device.

Claims

1. a laser scanner capable of scanning a laser extending in a direction parallel to a Y axis in an XYZ orthogonal coordinate system in a direction parallel to an X axis in the XYZ orthogonal coordinate system; a camera capable of scanning an imaging area extending in a direction parallel to the Y axis in a direction parallel to the X axis; a synchronization circuit for controlling the laser scanner and the camera; Equipped with the laser scanner and the camera are arranged along the X axis of the XYZ Cartesian coordinate system; the synchronization circuit controls the laser scanner and the camera so that the direction of irradiation of the laser emitted from the laser scanner and the line of sight direction of the camera intersect on a two-dimensional plane in a three-dimensional space; the camera captures an image of an object intersecting the two-dimensional plane by scanning in a −X direction of the X axis; the synchronization circuit synchronizes the scanning of the laser scanner and the camera in the −X direction to acquire an image of the object; the synchronization circuit acquires an image of the background of the object by making the shutter open time of the camera longer while the laser is being irradiated than when the scan in the −X direction is being performed; The synchronization circuit generates a composite image by combining the image of the object and the image of the background. An image sensing device characterized by:

2. 2. The image sensing device according to claim 1, wherein the shutter opening time is equal to or greater than one-fourth of the scanning time when generating the light curtain generated on the two-dimensional surface.

3. 2. The image sensing device of claim 1, wherein the image of the background is acquired by scanning the camera in the −X direction while the irradiation direction of the laser scanner moves in the +X direction of the X axis.