Three-dimensional sensing apparatus

Abstract

A three-dimensional sensing apparatus (1) has: an illumination device (10); an imaging unit (20) that captures an image of a light irradiation region; and a control unit (30) that controls the illumination device (10) and the imaging unit (20). The illumination device (10) emits light so as to form a light irradiation region that extends in a first direction, and moves the light irradiation region (100) in a second direction that intersects the first direction. The control unit (30) acquires a captured image for each of imaging regions (622) located at different positions in a third direction, and determines the three-dimensional position of an object (541) by using luminance information of pixels included in the same comparison region in the captured images.

Description

3D sensing device

[0001] The present disclosure relates to a three-dimensional sensing device.

[0002] In recent years, a system has been proposed that includes a light source, a line sensor, a processor, and software for creating a light sheet plane (also called a "light curtain"), and detects objects in an area defined by the intersection of the light sheet plane and the sensing plane of the line sensor (see, for example, Patent Document 1).

[0003] JP 2022-530349 A (see, for example, claim 1)

[0004] However, conventional techniques have a problem in that the distance resolution is low.

[0005] An object of the present disclosure is to solve the above-mentioned problems and to improve distance resolution.

[0006] A three-dimensional sensing device according to one aspect of the present disclosure is a three-dimensional sensing device that detects an object present in a target space, and includes: an illumination device that forms a light irradiation area extending in a first direction by irradiating light and moves the light irradiation area in a second direction that intersects with the first direction; an imaging unit that images the light irradiation area; and a control unit that controls the illumination device and the imaging unit so that an imaging area, which is an intersection range where a line imaging range imaged by the imaging unit intersects with the light irradiation area, moves in a third direction that is perpendicular to the first direction and the second direction and in the second direction, and the control unit acquires an image for each imaging area positioned differently in the third direction, and determines the three-dimensional position of the object using brightness information of pixels included in the same comparison area of ​​each of the imaged images.

[0007] According to the present disclosure, distance resolution can be improved.

[0008] 1 is a perspective view schematically illustrating the configuration of a three-dimensional sensing device (also referred to as a light curtain sensor device) for performing light curtain sensing in the present disclosure. FIG. 1 is a diagram illustrating an example of the configuration of a laser scanner. (A) to (C) are diagrams respectively illustrating a 3D image of an object, an image of the background of the object, and a composite image of these. FIG. 1 is a diagram illustrating the generation of a light curtain in the XZ plane. FIG. 2 is a diagram illustrating the operation of a light curtain on a virtual screen. FIG. 3 is a diagram illustrating the operation of a light curtain on a virtual screen. FIG. 4 is a diagram illustrating a control method for alternately acquiring sensing by a light curtain and a background image. FIG. 5 is a diagram illustrating a geometric arrangement for generating a light curtain. FIG. 6 is a diagram illustrating an example of the operation of a three-dimensional sensing device for measuring an object. FIG. 7 is a schematic diagram illustrating the configuration of a three-dimensional sensing device 1. FIG. 1A is a diagram illustrating the detection of a quadrangular pyramidal object by the three-dimensional sensing device, and FIG. 1B is a diagram illustrating the portion detected by the light curtain of FIG. 1A. FIG. 1C is a diagram illustrating the detection of a quadrangular pyramidal object by the three-dimensional sensing device moving in the depth direction. 1A is a diagram showing the results of a three-dimensional sensing device detecting a quadrangular pyramidal object using a light curtain moving in the depth direction, and FIG. 1B is a diagram showing the shape of the object obtained by integrating the results of FIG. 1A. A diagram showing how a light curtain is formed by an imaging area, which is an intersection area where a light irradiation area formed by laser light emitted from a laser scanner and moving in the X direction intersects with a line imaging area formed by an imaging unit and moving in the X direction. A diagram showing the luminance distribution detected by the three-dimensional sensing device for each positional relationship between a light curtain of thickness Tc and an object. A diagram showing frames with different distances in the Z direction, and FIG. 1B is a diagram showing the luminance values ​​for each frame. A diagram showing frames with different distances in the Z direction, and FIG. 1B is a diagram showing the luminance values ​​for each frame. A diagram showing the difference in thickness of the light curtain depending on the distance to the light curtain. A diagram showing the difference in thickness of the light curtain depending on the distance to the light curtain. A diagram showing the difference in thickness of the light curtain depending on the distance to the light curtain. A diagram showing the difference in distance from the three-dimensional sensing device to the object.1A to 1C are diagrams showing the difference in the number of frames depending on the distance from the three-dimensional sensing device to the object. 1A is a diagram showing that the thickness of the light curtain increases by increasing the exposure time of the imaging unit, and 1B is a diagram showing that the thickness of the light curtain decreases by shortening the exposure time of the imaging unit. 1A to 1C are diagrams showing the brightness values ​​of frames for each exposure time when an object is scanned while changing the distance from the three-dimensional sensing device to the light curtain. 1A is a diagram showing the position of the center of gravity based on the brightness values ​​of each frame. 1B is a diagram showing the position of the center of gravity based on the brightness values ​​of each frame. 1C is a diagram showing an example of differential processing between a scanned image and a background image. 1C is a diagram showing another example of differential processing between a scanned image and a background image. 1D is a diagram showing an example of brightness values ​​when multiple objects are present.

[0009] 2. Overview of Configuration and Operation of Three-Dimensional Sensing Device According to Embodiments <<Three-dimensional Sensing Device 1>> In the XYZ Cartesian coordinate system shown in each drawing, the Z direction (Z axis) indicates a direction perpendicular to the virtual screen, the Y direction (Y) indicates a direction perpendicular to the Z direction and parallel to the imaging area, and the X direction (X axis) indicates a direction perpendicular to both the Z direction and the Y direction. Note that the Z direction is an example of a third direction, the X direction is an example of a second direction, and the Y direction is an example of a first direction.

[0010] 1 is a perspective view showing a schematic configuration of a three-dimensional sensing device 1 (also referred to as a light curtain sensor device) for performing light curtain sensing in the present disclosure. The example shown in FIG. 1 shows how a light curtain 300 is generated by combining a laser scanner 10 and an imaging unit 20.

[0011] The three-dimensional sensing device 1 has a laser scanner 10 as an illumination device, an imaging unit 20, and a control unit 30. In the three-dimensional sensing device 1, the laser scanner 10 and the imaging unit 20 are arranged side by side along the X direction, and are controlled by the control unit 30.

[0012] A laser beam 512 (also referred to as a line laser) that spreads in the Y direction is emitted from the laser scanner 10. A virtual screen 510, which is a plane for the purpose of explanation, exists perpendicular to the Z direction at a position Z0 away from the laser scanner 10 in the Z direction. The laser beam emitted from the laser scanner 10 appears as a linear laser beam 512 that extends in the Y direction on the virtual screen 510.

[0013] <Laser Scanner 10> Fig. 2 is a diagram showing an example of the configuration of the laser scanner 10. The laser scanner 10 emits a laser. The laser scanner 10 can scan a laser extending in a direction parallel to the Y axis of an XYZ Cartesian coordinate system in a direction parallel to the X axis. The laser scanner 10 and the imaging unit 20 are arranged along the X axis of the XYZ Cartesian coordinate system. A light irradiation area is formed by the laser emitted from the laser scanner 10. The laser scanner 10 moves the light irradiation area in the X direction by scanning the laser in a direction parallel to the X axis (X direction). Note that the laser scanner 10 is an example of a lighting device.

[0014] A laser beam with a wavelength λ is emitted from a laser light source 505, and the laser beam is converted into a laser 512 (i.e., laser light) that spreads in the Y direction by a line beam generating element 506. The line beam generating element 506 is, for example, a cylindrical lens or a Powell lens. The laser 512 is deflected in the Z direction by a galvanometer mirror 507. The galvanometer mirror 507 can be swung around the Y axis at an angle of ±α, and the laser 512 is scanned around the Y axis within an angle range of ±2α. On the virtual screen 510, the laser 512 is scanned from the +X direction to the −X direction, and the entire area of ​​the scan range 513 in FIG. 1 or 2 is illuminated.

[0015] <<Image Capture Unit 20>> The image capture unit 20 can scan a line image capture range 200 (also referred to as a linear image capture range) extending in a direction parallel to the Y axis in an XYZ Cartesian coordinate system in a direction parallel to the X axis in the XYZ Cartesian coordinate system. For example, the image capture unit 20 captures an object 541 that intersects with a two-dimensional plane in three-dimensional space by scanning in the −X direction of the X axis. The image capture unit 20 captures the line image capture range 200 extending in a direction parallel to the Y axis (Y direction). The image capture unit 20 moves the line image capture range 200 in the X direction by scanning in a direction parallel to the X axis (X direction). Note that the image capture unit 20 is an example of an image capture unit.

[0016] In this embodiment, the imaging unit 20 is a rolling shutter camera. The imaging unit 20 reads out luminance information of the image sensor pixels in order, starting from the top row. By shortening the exposure time of the imaging unit 20 under the control of the control unit 30, a linear imaging area extending in the row direction of the image sensor can be scanned in the column direction perpendicular to the row direction. For example, in a 1000-row rolling shutter camera operating at a frame rate of 50 fps, if the exposure time per row is set to te = 20 μs, adjacent rows can be separated in time and sensing can be performed row by row. In actual operation, the control unit 30 controls the imaging unit 20 to operate such that the exposure area shifts in order of five rows, for example, with an exposure time of te = 100 μs.

[0017] Normally, a rolling shutter camera is used so that the row direction is horizontal, but in this embodiment, the imaging unit 20 is installed rotated -90° around the optical axis of the imaging unit. The imaging unit 20 is installed so that the optical axis direction of the imaging unit 20 faces slightly inward from the Z direction (-X direction) so that the entire imaging range 523 of the imaging unit 20 on the virtual screen 510 roughly overlaps with the entire laser scanning range 513. However, even if the imaging unit 20 faces the Z direction, if there is overlap between the entire imaging range 523 and the entire scanning range 513, the light curtain 300 is generated within the overlapping region.

[0018] The imaging unit 20 is also provided with a bandpass filter 508. The bandpass filter 508 may be installed at the tip of the lens barrel of the imaging unit 20 or inside the imaging unit 20. The bandpass filter 508 is a wavelength filter that passes only light of wavelength λ emitted from the laser light source 505.

[0019] <<Control Unit 30>> The control unit 30 controls the laser scanner 10 and the imaging unit 20. The control unit 30 can synchronize the operation of the laser scanner 10 and the operation of the imaging unit 20. For example, the control unit 30 controls the laser scanner 10 and the imaging unit 20 so that the irradiation direction of the laser emitted from the laser scanner 10 and the line of sight direction of the imaging unit 20 intersect on a two-dimensional plane in three-dimensional space. In other words, the control unit 30 controls the laser scanner 10 and the imaging unit 20 so that an irradiation area formed by the laser irradiation from the laser scanner 10 and a line imaging range 200 imaged by the imaging unit 20 intersect on a two-dimensional plane in three-dimensional space. Note that the intersection area where the irradiation area and the line imaging range 200 intersect is the imaging area.

[0020] 3A to 3C are diagrams showing a 3D image of an object 541, an image of the background of the object 541, and a composite image of these, respectively. The three-dimensional sensing device 1 can generate a composite image by acquiring a 3D image of the object 541 and an image of the background of the object 541 and combining these images. For example, as shown in FIG. 3A, the control unit 30 acquires an image of the object 541 by synchronizing scans (also referred to as first scans) in the −X direction of the laser scanner 10 and the image capture unit 20. That is, the control unit 30 controls the scanning in the −X direction of the laser scanner 10 and the image capture unit 20 so that the captured image area moves in the −X direction by synchronizing the scans in the −X direction of the laser scanner 10 and the image capture unit 20. As shown in FIG. 3B, the control unit 30 acquires an image of the background of the object by lengthening the shutter opening time (also referred to as exposure time) of the image capture unit 20 while the laser is irradiated compared to the time when the first scan in the −X direction was performed. The image of the background of the object 541 is also referred to as a background image. The background image is acquired, for example, by the laser scanner 10 and the imaging unit 20 scanning from the +X direction to the −X direction of the X axis.

[0021] The control unit 30 generates a composite image by combining the image of the object 541 with the background image. As shown in Fig. 3(C) , this processing enables the three-dimensional sensing device 1 to generate a composite image in which the 3D image of the object 541 is combined with the background image. In other words, the three-dimensional sensing device 1 is an image sensing device that can generate an image including an image of the sensing area and an image behind or around the sensing area.

[0022] <Light Curtain Sensing> Fig. 4 is a diagram illustrating the generation of the light curtain 300 in the XZ plane. Figs. 5 and 6 are diagrams illustrating the operation of the light curtain 300 on a virtual screen 510. In Fig. 4, the angle θ formed between the Z axis and the irradiation direction 511 of the laser 512 from the Z direction is defined as θ. Note that the clockwise direction is positive. In Fig. 4, the angle θa formed between the irradiation direction 511a and the Z axis at a certain time ta has a negative value. Similarly, the angle θb formed between the irradiation direction 511b and the Z axis at a time tb has a positive value. Note also that the positive direction of the X axis is to the left in the figure.

[0023] In FIG. 4 , the imaging unit 20 is installed rotated -90° around the optical axis, so that the line of sight 521 of the imaging unit 20 scans clockwise within the XZ plane. The angle with the Z axis is designated as φ clockwise. In FIG. 4 , the angle φa between the line of sight 521a and the Z axis at time ta, and the angle φb between the line of sight 521b and the Z axis at time tb are both positive values. Here, at time ta, the illumination direction 511a and the line of sight 521a intersect at an intersection 524a on the virtual screen 510a. Similarly, at time tb, the illumination direction 511b and the line of sight 521b intersect at an intersection 524b on the virtual screen 510a.

[0024] 5 and 6 show the relationship between the laser 512 and the line imaging range 200 on the virtual screen 510a at a distance Za and the virtual screen 510b at a distance Zb. On the virtual screen 510a, the laser 512 and the line imaging range 200 are scanned so as to always overlap. In other words, the overlap between the laser 512 and the line imaging range 200 forms an imaging region, which is an intersection range where the laser 512 and the line imaging range 200 intersect. As will be described later, such operation is possible by appropriately controlling the angle of the galvanometer mirror 507 with respect to time. At this time, the virtual screen 510b is installed at a different distance from the virtual screen 510a, and therefore, as is clear from FIG. 6, the laser 512 and the line imaging range 200 do not overlap.

[0025] When the light curtain 300 is generated on the virtual screen 510a and the light curtain 300 intersects with an object, an image other than the intersecting area between the light curtain 300 and the object is not captured by the imaging unit 20. This is because the exposure time of each pixel in one frame is extremely short (for example, exposure time te = 60 μs), and the bandpass filter 508 cuts off light rays other than those with the laser wavelength before they reach the image sensor inside the imaging unit 20. Therefore, the background is not captured by light curtain sensing.

[0026] Here, the conditions for scanning the laser 512 and the line imaging range 200 so that they overlap on the virtual screen 510a will be described. First, when the laser 512 is linear and extends in the Y direction, the line imaging range 200 of the imaging unit 20 must extend accurately in the Y direction. To achieve this, a fine adjustment mechanism is provided for finely adjusting the orientation of the imaging unit 20 around the optical axis. Furthermore, the imaging lens of the imaging unit 20 must be a distortion-corrected lens. Otherwise, the line imaging range 200 will be distorted from its linear shape. Furthermore, to scan the laser 512 and the line imaging range 200 while keeping them overlapping on the virtual screen 510a, the timing at which the imaging unit 20 starts imaging and the timing at which the laser scanner 10 starts scanning must be synchronized with an appropriate delay, and the control unit 30 is used for this purpose. Furthermore, the scanning speed of the laser 512 and the scanning speed of the line imaging range 200 must be the same on the virtual screen 510.

[0027] 7 is a diagram illustrating a control method for alternately capturing sensing using a light curtain and background images. In FIG. 7, the dashed line represents the angular function φ(t) of the line-of-sight direction of the image capture unit 20 in the line capture range, and the solid line represents the angular function θ(t) of the irradiation direction 511 of the laser 512. In sections P1 and P3, a planar light curtain perpendicular to the Z axis is generated at a position at a different distance Z0, and background images are captured in sections P2 and P4. The specific parameters used to calculate the graph shown in FIG. 7 will be described later.

[0028] First, the operation in section P1 will be described. Figure 8 is a diagram showing the geometric arrangement for generating the light curtain 300. An irradiation direction 511 of a laser 512 and a line of sight direction 521 of the image capturing unit 20 intersect on a virtual screen 510a placed at a distance Z0 in the Z direction. As shown in Figure 8, the angular function θ(t) of the irradiation direction 511 of the laser 512 and the angular function φ(t) of the line of sight direction 521 of the image capturing unit 20 are angles from the Z direction. In the arrangement shown in Figure 8, both the angle θ and the angle φ have positive values. The angle of the galvanometer mirror is half this angle (α = θ(t) / 2). The angular function φ(t) of the line of sight direction of the image capturing unit 20 is expressed by the following equation (1) using a constant coefficient k determined by the frame rate: φ(t) = k × t + φmin (1)

[0029] Since it is generally difficult to operate a rolling shutter camera at a line scan speed other than a constant speed, the angular function θ(t) of the irradiation direction 511 of the laser 512 is controlled to match the angular function φ(t) of the line of sight direction of the image capture unit 20. Using Figure 8, the relationship between θ(t) and φ(t) is found. The rotation axis of the laser scanner 10 is set to point Q, the rotation axis of the line of sight direction 521 of the image capture unit 20 is set to point R, and the distance between them is set to b. The foot of the perpendicular line drawn from the intersection point 524 to the line QR is set to point P. Then, the signed distance PR is expressed as Z0 * tan φ using φ. Furthermore, since the signed distance PQ is expressed as Z0 * tan θ, the following equation (2) holds. Z0 × tan φ = Z0 × tan θ + b (2)

[0030] That is, when the angle φ is fixed, the angle θ of the laser 512 is calculated by the following equation (3): θ=Arctan(tan φ−b / Z0) (3)

[0031] When the virtual screen 510a is a plane perpendicular to the Z axis, the distance Z0 is a constant value. The function that generates the planar light curtain 300 at the distance Z0 is shown in the graph of section P1 in Figure 7. The specific parameters for plotting the angle function for section P1 are as follows:

[0032] <<Parameters for Section P1>> Scanning range of laser 512: -25°≦θ≦25° (θmin=-25°, θmax=25°) Scanning range of imaging unit 20: -15°≦φ≦35° (φmin=-15°, φmax=35°) b=0.15 m Time required for imaging unit 20 to scan all lines T=16 ms Z0=0.5 m The coefficient k in equation (1) is expressed by the following equation (4): k=(φmax-φmin) / T (4) Here, imaging unit 20 has a field angle of 50° in the XZ plane, and its optical axis is tilted by +10° with respect to the Z direction.

[0033] The position of the plane with parameters Z0 = 0.5 m in section P1 corresponds to the position marked by virtual screen 510b in Figure 4. In the graph of section P1 shown in Figure 7, the angular function φ(t) of the line of sight direction of the imaging unit 20 varies linearly with time t, as expressed by equation (1), and varies within the range of φmin = -15° to φmax = 35° between time t = 0 and 16 ms. The angle θ of the laser 512 during this period is given by equation (3), which is shown in the diagram as the angular function θ(t) of the irradiation direction 511 of the laser 512 in section P1(1). θ and φ each have limited scanning ranges, and a light curtain is generated only when both θ and φ are within their scanning ranges. In other words, the light curtain generation region is between point 526 and point 527 in Figure 4.

[0034] The intersection of the irradiation direction of the laser 512 at the minimum angle θmin and the virtual screen 510b is designated as 526, and the intersection of the line of sight direction 521 at the maximum angle φmax of the imaging unit 20 and the virtual screen 510b is designated as 527.

[0035] When the laser scan in section P1(1) is completed, the angle of the galvanometer mirror 507 must be returned in the negative direction in preparation for the next laser scan. Because the galvanometer mirror 507 is an object with a moment of inertia, a non-small finite time Tback is required to return it to the start position of the next scan. While this differs depending on the product specifications of the galvanometer mirror 507, for example, Tback = 4 ms is required. In section P1(2), the angle of the laser 512 is swung from +25° to -25° using a sine function.

[0036] <Section P3> In section P3, the light curtain 300 is set at a position at a distance Z0 = 2 m, which is different from that in section P1. The parameters other than Z0 are the same as those in section P1.

[0037] <<Operation of Control Unit 30>> The control unit 30 is used to operate the angular function φ(t) of the line of sight direction of the imaging unit 20 and the angular function θ(t) of the irradiation direction 511 of the laser 512 at the precise timing shown in FIG. 7. The control unit 30 is pre-programmed with a function waveform that provides the angular function θ(t) of the irradiation direction 511 of the laser 512 and the timing for outputting a trigger pulse shown in FIG. 7 in accordance with the angular function θ(t). The control unit 30 outputs an analog voltage waveform that provides the angular function θ(t) to the laser scanner 10. A trigger pulse is output to the imaging unit 20. The imaging unit 20 starts capturing one frame of image data in response to the rising edge of the trigger pulse voltage. In other words, it starts scanning the line imaging range.

[0038] <3D measurement using 3D sensing device 1> In Figure 7, a planar light curtain 300 was generated at positions Z0 = 0.5 m and 2 m in sections P1 and P3, respectively. However, 3D measurement can be performed by narrowing the spacing between the light curtain surfaces in the Z direction and installing multiple light curtains.

[0039] 9 is a diagram showing an example of the operation of the three-dimensional sensing device 1 for measuring the position of an object 541. In FIG. 9, in a section Pn, a light curtain is installed at a distance Zn calculated by the following equations (5) and (6). That is, the control unit 30 controls the imaging area, which is the intersection area where the line imaging area 200 imaged by the imaging unit 20 and the light irradiation area of ​​the laser 512 intersect, to move in the X direction and the Z direction, thereby installing the light curtain at the distance Zn. n=1, 2, ..., N (N is a natural number) (5) Zn=Z0+(n-1)ΔZ (6) (n is a natural number)

[0040] That is, the distance between adjacent light curtains is ΔZ. An object 541 is placed within the sensing area of ​​such a light curtain. Images of cross sections of the object 541 where the light curtains 300a, 300b, 300c, ... intersect are captured by the imaging unit 20. The cross-sectional area of ​​the object 541 reflected by the light curtain at a distance Zn is at a distance Zn from the three-dimensional sensing device 1. Since many cross-sectional images at different distances Zn are acquired, distance information of the surface of the object 541, i.e., a depth map, can be obtained.

[0041] 10 is a schematic diagram showing the configuration of a three-dimensional sensing device 1. The three-dimensional sensing device 1 is an image sensing device (i.e., a vision sensor) that captures an object present on the surface of a curtain-shaped light curtain 300, but does not capture an object that is not present on the surface of the light curtain 300. The three-dimensional sensing device 1 can also detect an object 541 that is present in a target space by moving the light curtain 300. "Detecting an object 541" means detecting the presence or absence of the object 541, or detecting the three-dimensional shape of the object 541, or both.

[0042] For example, the three-dimensional sensing device 1 can detect whether or not a person, as an object 541, is present in a room, as a target space, by moving the light curtain 300. Furthermore, the three-dimensional sensing device 1 can detect the three-dimensional shape of an object, such as an article or equipment, as the object 541 present in the target space by moving the light curtain 300.

[0043] The three-dimensional sensing device 1 includes a laser scanner 10 as an illumination device, an imaging unit 20, and a control unit 30.

[0044] The laser scanner 10 is a device that forms a light irradiation area 100 extending in a first direction by irradiating light and moves the light irradiation area 100 in a second direction that intersects with the first direction. In other words, the laser scanner 10 is an optical scanning device that can scan light (e.g., laser light) in the first direction and the second direction.

[0045] The first direction is, for example, the Y direction parallel to the Y axis in an XYZ Cartesian coordinate system in the target space, and is the height direction or the extension direction of the light irradiation area 100 in FIG. 10. The first direction is also the extension direction of the line imaging range 200 in FIG. 10. The second direction is, for example, the X direction parallel to the X axis in the XYZ Cartesian coordinate system, and is the horizontal direction or the movement direction D100 of the light irradiation area 100 in FIG. 10. The second direction is also the movement direction D200 of the line imaging range 200. Although FIG. 10 shows a case where the first direction and the second direction are orthogonal to each other, the first direction and the second direction do not necessarily have to be orthogonal to each other.

[0046] The laser scanner 10 is configured, for example, from a laser light source as a light source and a galvanometer mirror (i.e., a galvanometer scanner) as a light scanning optical system (scanning mirror). However, the configuration of the laser scanner 10 is not limited to this.

[0047] The imaging unit 20 captures an image of the light irradiation area 100. The imaging unit 20 is a device that performs an imaging operation to capture an image of a line imaging range 200, which is a linear imaging range extending in a first direction, and moves the line imaging range 200 in a second direction. In other words, the imaging unit 20 is a device that moves the line imaging range 200 in the second direction every time an image is captured in the line imaging range 200.

[0048] The imaging unit 20 is a sensor in which a plurality of imaging pixels are arranged in a line or two-dimensionally. The imaging unit 20 is, for example, a line sensor or a camera. The camera is, for example, a rolling shutter camera.

[0049] The control unit 30 controls the laser scanner 10 and the imaging unit 20 so that an imaging area 622, which is an intersection area where the line imaging area 200 imaged by the imaging unit 20 intersects with the light irradiation area 100, moves in a third direction orthogonal to the first direction and the second direction, and in the second direction. The third direction is the Z direction.

[0050] 10 , the control unit 30 controls the laser scanner 10 and the imaging unit 20 to form a light curtain 300 made up of an imaging area 622, which is an intersection area where the light irradiation area 100 and the line imaging range 200 intersect, and detects an object (for example, an object 541 in FIG. 11 , which will be described later) that overlaps the light curtain 300 based on an image captured by the imaging unit 20. The light curtain 300 is a collection of imaging areas 622 and is also called a "projection-type light curtain."

[0051] The control unit 30 controls the lighting device (e.g., the laser scanner 10) and the imaging unit 20 to form a light curtain, which is a collection of imaging areas 622. The control unit 30 can make the light curtain 300 a virtual plane or a virtual curved surface.

[0052] The control unit 30 changes the position of the light curtain 300 in the depth direction (i.e., the depth direction) corresponding to the distance from the imaging unit 20 by controlling the light irradiation direction of the laser scanner 10 and the line of sight direction (i.e., the imaging direction) of the imaging unit 20. In Fig. 10 , the depth direction is the Z direction. The control unit 30 can also change the thickness of the light curtain 300 in the depth direction by controlling the exposure time of the imaging unit 20 (i.e., the open time of the camera shutter).

[0053] The control unit 30 acquires captured images for each of the imaging areas 622, each of which has a different position in the third direction. Each of the imaging areas 622 has a length in the second direction and a third direction (i.e., a width in the imaging area). The control unit 30 determines the three-dimensional position of an object (e.g., object 541 in FIG. 11 , described below) using brightness information of pixels included in the same comparison area in each of the multiple captured images. Here, the comparison area may be an area formed by any one pixel in the captured image, or an area formed by a set of any one pixel and its surrounding pixels (several to several dozen pixels). Furthermore, the same comparison area in each of the captured images means that the comparison areas set in each of the captured images are located at the same positions on the captured images.

[0054] For example, the control unit 30 uses the brightness information to determine the position of the object in the third direction based on the captured image having the maximum brightness among pixels included in the same comparison region of each of the multiple captured images. In other words, the control unit 30 uses the brightness information to determine the captured image having the maximum brightness pixel as the position of the object in the third direction at that pixel based on the captured image having the maximum brightness among pixels included in the same comparison region of each of the multiple captured images. Then, the control unit 30 determines the position of the object in the third direction based on the captured image having the maximum brightness among pixels included in the same comparison region of each of the multiple captured images, at least for pixels in which the object appears in the captured image. This can improve the distance resolution.

[0055] The control unit 30 is configured, for example, by a control circuit such as a synchronization circuit. The control circuit may have a memory as a storage device for storing a software program and a processor such as a CPU (Central Processing Unit) for executing the program. The control unit 30 may also be configured by a dedicated processing circuit or a general-purpose computer.

[0056] <3D Measurement by the 3D Sensing Device 1> FIG. 11A is a diagram illustrating the detection of a quadrangular pyramid-shaped object 541 by the 3D sensing device 1. FIG. 11B is a diagram illustrating a portion 310 detected by the light curtain 300 of FIG. 11A. As shown in FIG. 11A, the light curtain 300 of the 3D sensing device 1 has a thickness Tc. The 3D sensing device 1 detects a portion 310 of the object 541 within an imaging area, which is an intersection range with the light curtain 300. The control unit 30 can change the thickness Tc of the light curtain 300 by controlling the light irradiation direction of the laser scanner 10 and the line of sight direction (i.e., the imaging direction) of the imaging unit 20. The control unit 30 can also change the thickness Tc of the light curtain 300 by adjusting the exposure time te of the imaging unit 20.

[0057] 12 is a diagram showing how the three-dimensional sensing device 1 detects a quadrangular pyramidal object 541 using light curtains 300a to 300e that move in the depth direction (Z direction). The light curtains 300a to 300e shown in FIG. 12 show how the light curtain 300 in FIG. 11 moves in the depth direction.

[0058] 12 , the shape of object 541 intersecting light curtain 300a is detected, then the shape of object 541 intersecting light curtain 300b is detected, and then the shapes of objects 541 intersecting light curtains 300c, 300d, and 300e are detected in order. As shown in FIG. 12 , the control unit 30 controls the laser scanner 10 and the imaging unit 20 to change the position of light curtain 300 in the depth direction.

[0059] Fig. 13(A) shows the results of detection by the three-dimensional sensing device 1 using a light curtain moving in the depth direction on a quadrangular pyramidal object 541, and Fig. 13(B) shows the shape of the object obtained by integrating the results of Fig. 13(A). Fig. 13(A) shows a front view and a perspective view of the results detected by the light curtains 300a, 300c, 300e, etc. shown in Fig. 12. Fig. 13(B) shows the integrated detection result 320.

[0060] 14 is a diagram showing how the light curtain 300 is formed by an imaging area 622, which is an intersection area between a light irradiation area 100 formed by laser light emitted from the laser scanner 10 and moving in the X direction, and a line imaging range 200 formed by the imaging unit 20 and moving in the X direction. As shown in FIG. 14 , during the exposure time te (i.e., the shutter open time) when the imaging unit 20 captures an image of one line of multiple imaging pixels aligned in the Y direction, the light irradiation area 100 is scanned, and a light curtain 300 having a thickness Tc is formed in the overlapping portion between the moving light irradiation area 100 and the line imaging range 200. In other words, the longer the exposure time te when the imaging unit 20 captures an image of one line of multiple imaging pixels aligned in the Y direction, the greater the movement distance of the light irradiation area 100 moving in the movement direction D100 during the exposure time te. This increases the overlapping portion between the light irradiation area 100 and the line imaging range 200, thereby increasing the thickness Tc of the light curtain 300. In the example of FIG. 14, the laser scanner 10 and the imaging unit 20 are arranged on a straight line extending in the X direction with a base length b between them.

[0061] <Brightness distribution during three-dimensional measurement> As described above, the imaging range of the light curtain has a thickness in the depth direction, so when the light curtain is scanned in the depth direction at intervals less than the thickness of the object 541 in order to obtain its three-dimensional shape as shown in Figure 12 and the number of images obtained is the number of scans, if one focuses on the pixels where the object is detected, brightness values ​​greater than the threshold value resulting from the reflected light of the object will appear in multiple consecutive images.

[0062] FIG. 15 shows the luminance distribution detected by the three-dimensional sensing device 1 for each positional relationship between the light curtain 300 having a thickness Tc and the object 541. The distance Z is the distance from the imaging unit 20 to each frame. In other words, the distance Z is the distance from the imaging unit 20 to the virtual screen on which the light curtain 300 is formed. If the surface of the object 541 is outside the thickness of the light curtain 300, the object 541 is not detected. If the surface of the object 541 falls within the thickness of the light curtain 300, reflected light from the surface of the object 541 is detected and the luminance increases. At this time, the luminance value due to the reflected light within the light curtain 300 is not constant; the luminance increases near the center of the thickness of the light curtain 300, and a luminance peak appears at a certain scan position.

[0063] <Distance calculation using luminance distribution> Fig. 16(A) is a diagram showing frames (i.e., captured images) with different distances in the Z direction. Fig. 16(B) is a diagram showing luminance values ​​for each frame. Fig. 17(A) is a diagram showing frames (i.e., captured images) with different distances in the Z direction. Fig. 17(B) is a diagram showing luminance values ​​for each frame. The distance Z in the Z direction for a pixel in row i and column j where a luminance value equal to or greater than a threshold is measured in multiple frames is calculated. ｉｊ To identify the distance, the frame with the maximum brightness value may be selected and converted into distance, as shown in Fig. 16(B). Alternatively, as shown in Fig. 17(B), brightness values ​​at multiple points may be interpolated to obtain a curve of brightness values, and the brightness peak position of the maximum brightness may be calculated with a resolution equal to or smaller than the scan interval. Note that one frame refers to an image obtained by scanning while moving the line imaging range 200 in the X direction.

[0064] Also, the distance z corresponding to each k-th frame ｋ and the kth frame luminance value M ｋ The center of gravity may be calculated using the following equation, and the distance to the pixel may be calculated to calculate the position of the luminance peak: ｉｊ = (ΣM ｋ xz ｋ ) / (ΣM ｋ )

[0065] Before performing the interpolation process or the calculation of the center of gravity, a moving average filter or a Gaussian filter may be applied to the brightness of each frame to remove noise components before calculating the distance. By performing the above process on all pixels, even if the light curtain 300 has a thickness, 3D shape data of the area can be acquired with a resolution equal to or less than the scan interval.

[0066] 18 and 19 are diagrams showing differences in the thickness of the light curtain 300 depending on the position in the depth direction (Z direction) of the light curtain 300. In other words, the thickness of the light curtain 300 differs depending on the distance Z from the three-dimensional sensing device 1 to the light curtain 300. As shown in Fig. 18, when the position from the three-dimensional sensing device 1 to the light curtain 300 is far, even if the laser rotates by the same angle for the same exposure time of the image capturing unit 20, the angle between the laser projection direction and the line of sight direction of the image capturing unit 20 becomes smaller, and the thickness of the light curtain 300 becomes larger.

[0067] 19 , when the position of the light curtain 300 is close to the three-dimensional sensing device 1, the angle between the laser projection direction and the line of sight direction of the image capturing unit 20 increases, resulting in a decrease in the thickness of the light curtain 300. Furthermore, the smaller the distance between the laser scanner 10 and the image capturing unit 20, the greater the thickness of the light curtain 300, and vice versa.

[0068] 20(A) to 20(C) are diagrams showing differences in the distance from the three-dimensional sensing device 1 to the object 541. In the examples shown in FIGS. 20(A) to 20(C), the difference in distance Z is Z1<Z2<Z3. FIGS. 21(A) to 21(C) are diagrams showing differences in the number of frames due to differences in distance Z from the three-dimensional sensing device 1 to the object 541. FIG. 21(A) corresponds to the distance Z1 shown in FIG. 20(A), FIG. 21(B) corresponds to the distance Z2 shown in FIG. 20(B), and FIG. 21(C) corresponds to the distance Z1 shown in FIG. 20(C). As shown in Figures 21(A) to 21(C), when object 541 is scanned while changing the distance to light curtain 300, the shorter the distance from three-dimensional sensing device 1 to object 541, the fewer frames in which brightness values ​​above the threshold value appear, and the longer the distance from three-dimensional sensing device 1 to object 541, the more frames in which brightness values ​​above the threshold value appear.

[0069] When determining the distance to an object from the brightness value, the closer the distance, the smaller the measurement variance, and the farther the distance, the larger the measurement variance. To measure with high accuracy, it is recommended to measure at a short distance from the 3D sensing device 1. Alternatively, the distance measurement accuracy of the light curtain 300 can be improved by increasing the distance between the laser scanner 10 and the imaging unit 20.

[0070] As shown in FIGS. 21A to 21C , the control unit 30 can determine the distance Z (i.e., the position in the third direction) of the object 541 based on a curve obtained from the brightness values ​​of pixels included in the same comparison region in each image of multiple frames using the brightness information. That is, the control unit 30 uses the brightness information to perform brightness value interpolation processing at multiple points as shown in FIG. 17B above, using the brightness values ​​of pixels included in the same comparison region in each of the multiple captured images, to obtain a brightness value curve. Based on the peak of the obtained curve, the control unit 30 determines the position of the peak as the position of the object in the third direction. Then, for at least pixels in which the object appears in the captured image, the control unit 30 obtains a brightness value curve as described above using the brightness values ​​of pixels included in the same comparison region in each of the multiple captured images, and determines the position of the object in the third direction based on the curve. This improves distance resolution. When obtaining the above-mentioned brightness curve from the brightness values ​​of pixels included in the same comparison area of ​​each of a plurality of captured images, if the comparison area includes a plurality of pixels, the brightness value of any one of the pixels included in the comparison area may be used as the representative brightness value, or the average brightness value of the plurality of pixels included in the comparison area may be used. Furthermore, when the brightness value of any one of the pixels included in the comparison area is used as the representative brightness value, the maximum brightness value of the brightness values ​​of the pixels included in the comparison area may be used.

[0071] Effect of Exposure Time on Luminance Distribution Figure 22(A) is a diagram showing that the thickness Tc of the light curtain 300 increases when the exposure time te of the imaging unit 20 is increased. Figure 22(B) is a diagram showing that the thickness of the light curtain 300 decreases when the exposure time te of the imaging unit 20 is decreased. Figures 22(A) and 22(B) show that the control unit 30 can change the thickness Tc of the light curtain 300 by adjusting the exposure time te of the imaging unit 20. In other words, the longer the exposure time te when the imaging unit 20 captures an image of one line of multiple imaging pixels arranged in the Y direction, the greater the movement distance of the light irradiation region 100 that moves in the movement direction D100 during the exposure time te. This increases the overlapping portion between the light irradiation region 100 and the line imaging range 200, thereby increasing the thickness Tc of the light curtain 300.

[0072] 23(A) to 23(C) show the luminance values ​​of frames for each exposure time when scanning an object 541 while changing the distance from the 3D sensing device 1 to the light curtain 300. In the examples shown in FIGS. 23(A) to 23(C), the difference in exposure time is T1<T2<T3. As shown in FIGS. 23(A) to 23(C), the shorter the exposure time, the fewer frames with luminance values ​​above the threshold, and the longer the exposure time, the more frames with luminance values ​​above the threshold. When determining the distance to an object from luminance values, the shorter the exposure time, the smaller the measurement variance, and the longer the exposure time, the greater the measurement variance. For high-accuracy measurements, it is recommended to shorten the exposure time. However, shortening the exposure time reduces the maximum luminance value, and setting it too short may result in the inability to detect reflected light from the object. Furthermore, the S / N ratio decreases relative to the thermal noise and readout noise of the image sensor, potentially reducing the accuracy of distance calculations. Therefore, it is recommended to set an appropriate exposure time. Even in such a case, the control unit 30 uses the luminance information to perform luminance value interpolation processing at multiple points as shown in FIG. 17B above, using the luminance values ​​of pixels included in the same comparison region in each of the multiple captured images, to obtain a luminance value curve. Based on the peak of the obtained curve, the control unit 30 determines the position of the peak as the position of the object in the third direction. Then, for a comparison region that includes at least pixels in which the object appears in the captured images, the control unit 30 obtains a luminance value curve as described above using the luminance values ​​of pixels included in the same comparison region in each of the multiple captured images, and determines the position of the object in the third direction based on the curve. This allows for improved distance resolution.

[0073] <<Effect of Offset Luminance Value>> Figures 24 and 25 are diagrams showing the center of gravity position based on the luminance value of each frame. Figure 26 is a diagram showing an example of differential processing between a scanned image (i.e., a captured image of an object) and a background image. If the camera's black level is set high as in Figures 24 and 25, a constant luminance value is output even when there is no reflected light from the object, and this is added to all frames as an offset luminance value. In this case, if the center of gravity position is calculated using the luminance values ​​of all frames, it will be shifted from the luminance peak. In this case, the luminance peak can be accurately calculated by subtracting the luminance value of the corresponding pixel in the background image acquired with the laser turned off from the luminance value of each frame, as shown in Figure 26, and setting the luminance outside the peak to 0 when calculating the center of gravity.

[0074] Figure 27 is a diagram showing another example of subtraction processing between a scanned image and a background image. The luminance value of a pixel in the background image may not match the luminance value of frames other than the peak luminance of each frame, resulting in the luminance of frames other than the peak luminance not being 0, as shown in Figure 27. It is recommended that the luminance value to be subtracted be set higher than the actual luminance value of the background image, and that the luminance values ​​of frames other than the peak luminance be set to 0. In this case, if the subtraction value is set too high, there is a risk that peaks with low luminance values ​​will be erased. Therefore, for example, in the case of an 8-bit output camera, it is recommended to set the value about two levels higher.

[0075] <<Multiple Brightness Peaks>> Figure 28 is a diagram showing an example of brightness values ​​when multiple objects are present. When scanning with the light curtain 300 to perform three-dimensional measurement, if multiple objects are present and moving during scanning, or if there is a translucent object in front of the object, multiple brightness peaks may appear as shown in Figure 28. In this case, the brightness peak positions of both peaks can be calculated, and the respective distances Z can be calculated. As a result, when there are multiple brightness peaks obtained using the brightness information, the control unit 30 can calculate the three-dimensional object positions corresponding to each of the multiple peaks.

[0076] For example, the frame with the maximum brightness distribution value among all frames may be acquired and converted into distance. Alternatively, the center of gravity may be calculated locally using the brightness values ​​of several frames surrounding the maximum brightness value frame, and the distance Z may be calculated with a resolution equal to or less than the scan interval.

[0077] Although the preferred embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the claims.

[0078] Various aspects of the present disclosure are described below as appendices. (Appendix 1) A three-dimensional sensing device for detecting an object present in a target space includes: an illumination device that forms a light-illuminated area extending in a first direction by emitting light and moves the light-illuminated area in a second direction intersecting the first direction; an imaging unit that images the light-illuminated area; and a control unit that controls the illumination device and the imaging unit so that an imaging area, which is an intersection area where a line imaging range imaged by the imaging unit intersects with the light-illuminated area, moves in a third direction orthogonal to the first and second directions and in the second direction, wherein the control unit acquires captured images for each imaging area positioned differently in the third direction and determines the three-dimensional position of the object using luminance information of pixels included in the same comparison area of ​​each of the captured images. (Appendix 2) The three-dimensional sensing device according to Appendices 1, wherein the imaging area has lengths in the second direction and the third direction. (Supplementary Note 3) The three-dimensional sensing device according to Supplementary Note 1 or 2, wherein the control unit uses the brightness information to determine the position of the object in the third direction based on the captured image having the maximum brightness at pixels included in the same comparison region of each of the plurality of captured images. (Supplementary Note 4) The three-dimensional sensing device according to any one of Supplements 1 to 3, wherein the control unit uses the brightness information to determine the position of the object in the third direction based on a curve obtained from brightness values ​​at pixels included in the same comparison region of each of the plurality of captured images. (Supplementary Note 5) The three-dimensional sensing device according to Supplementary Note 4, wherein the control unit determines the position of the object in the third direction based on a brightness peak in the curve, and if the curve has multiple peaks, determines each of the multiple peaks as the position of the object in the third direction. (Supplementary Note 6) The three-dimensional sensing device according to any one of Supplements 1 to 5, wherein the control unit forms a light curtain, which is a collection of the captured regions, by controlling the lighting device and the image capture unit.(Supplementary Note 7) The three-dimensional sensing device according to any one of Supplementary Notes 1 to 6, wherein the imaging unit is a line sensor or a rolling shutter camera.

[0079] 1 Three-dimensional sensing device, 10 Laser scanner, 20 Imaging unit, 30 Control unit, 100 Light irradiation area, 200 Line imaging range, 300 Light curtain, 506 Line beam generating element, 507 Galvanometer mirror, 508 Bandpass filter, 510 Virtual screen, 511 Irradiation direction, 512 Laser, 513 Scan range, 541 Object, 622 Imaging area.

Claims

1. A three-dimensional sensing device that detects objects present in a target space, comprising: an illumination device that forms a light-illuminated area extending in a first direction by irradiating light and moves the light-illuminated area in a second direction that intersects with the first direction; an imaging unit that images the light-illuminated area; and a control unit that controls the illumination device and the imaging unit so that an imaging area, which is an intersection range where the line imaging range imaged by the imaging unit intersects with the light-illuminated area, moves in a third direction that is perpendicular to the first and second directions and in the second direction, wherein the control unit obtains an image for each imaging area positioned differently in the third direction, and determines the three-dimensional position of the object using brightness information for pixels included in the same comparison area of ​​each of the imaged images.

2. The three-dimensional sensing device according to claim 1, wherein the imaging area has lengths in the second direction and the third direction.

3. The three-dimensional sensing device described in claim 1 or 2, characterized in that the control unit uses the brightness information to determine the position of the object in the third direction based on the captured image having the maximum brightness in pixels included in the same comparison area of ​​each of the multiple captured images.

4. A three-dimensional sensing device as described in any one of claims 1 to 3, characterized in that the control unit determines the position of the object in the third direction based on a curve obtained from the brightness values ​​of pixels included in the same comparison area of ​​each of the multiple captured images using the brightness information.

5. The three-dimensional sensing device described in claim 4, characterized in that the control unit determines the position of the object in the third direction based on a brightness peak in the curve, and if there are multiple peaks in the curve, determines each of the multiple peaks as the position of the object in the third direction.

6. A three-dimensional sensing device according to any one of claims 1 to 5, wherein the control unit forms a light curtain, which is a collection of the imaging areas, by controlling the lighting device and the imaging unit.

7. A three-dimensional sensing device according to any one of claims 1 to 6, wherein the imaging unit is a line sensor or a rolling shutter camera.

Images