Three-dimensional measuring device

Abstract

A three-dimensional measurement device includes: a light projecting unit that projects reference light toward an object while scanning; a light receiving unit that receives the reference light reflected by the object; a time range setting unit that sets a scanning time range for the reference light corresponding to a distance measurement range for each predetermined section of the light receiving unit; and a three-dimensional information calculating unit that calculates three-dimensional information of the object (W) by triangulating information from the light receiving unit within the set scanning time range.

Description

Technical Field

[0001] The present invention relates to three-dimensional measurement technology, and in particular to an optical scanning three-dimensional measurement device. Background Art

[0002] As distance measurement methods based on triangulation using structured illumination, light projection methods such as the light sectioning method, the phase shifting method, and the spatial encoding method have been proposed. In the light sectioning method, a strip of slit light is projected onto the object while scanning it, and the object is photographed from a shooting position different from the projection position. Based on the projection angle of the slit light, the angle of incidence of the slit light on the shooting surface, and the baseline length between the projection position and the shooting position, the distance to the object is calculated by triangulation. The projection angle of the slit light is calculated, for example, based on the command value to the scanner and the detection time of the bright line of the slit light that appears on the shooting surface, and the incident angle of the slit light is calculated, for example, based on the incident position of the slit light on the shooting surface. The light sectioning method is considered to have good accuracy, but compared with the phase shifting method, the spatial encoding method, etc., the number of images required for one measurement increases, so there is a problem of taking up too much time for the measurement.

[0003] In recent years, an event-based image sensor has been proposed, which is different from the general frame-based image sensor. A frame-based image sensor opens and closes the shutter for a predetermined time to perform exposure, thereby outputting a frame image at a predetermined period. In contrast, an event-based image sensor independently and asynchronously monitors each pixel at all times. When an event is detected (for example, a brightness change greater than a predetermined value), the position, time, and polarity of the pixel with the event (for example, whether it becomes brighter (positive polarity) or darker (negative polarity)) are output as event information. Compared to frame-based image sensors, event-based image sensors have a wider dynamic range and only output event information, so they are characterized by high speed. Therefore, it is believed that the use of event-based image sensors can achieve high-speed light-cutting methods.

[0004] While ideally, events generated by an event-based image sensor would be solely those caused by the scanning of slit light, capturing an actual scene might not necessarily result in only ideal events; noise events may also occur due to factors other than slit light scanning. For example, in distance measurement based on the light sectioning method, there's the issue of multiple reflections of slit light. When focusing on a pixel in an image sensor, the intended target is to receive only the single reflection of slit light directed toward a specific portion of the object reflected in that pixel. However, sometimes, single reflections of slit light directed toward other portions of the object are reflected twice at that specific portion and enter that pixel. If this occurs, the pixel cannot distinguish whether it captured the slit light directly reflected (single reflection) from that specific portion or captured multiple reflections, making it impossible to measure the correct distance. Multiple reflections are light that has been reflected multiple times, so if any reflective surface has a reflectivity that isn't 100%, it will be weaker than single reflections. Therefore, while frame-based image sensors use brightness information to mitigate the effects of multiple reflections of reference light, event-based image sensors don't output brightness information, so the same approach can't be used. Furthermore, noise is not only caused by reference light reflected multiple times on an object, but also includes noise caused by the optical system and image sensor (eg, flare, ghosting, aberration, etc.) As a technology related to the present application, for example, the literature described below is well known.

[0005] Patent document 1 discloses a three-dimensional input device that uses slit light to scan an object while synchronously capturing images using two image sensors, stores the frame number at which the maximum brightness is detected for each pixel of each image sensor, matches pixels based on the frame number, and calculates distance based on the matched pixels.

[0006] Patent document 2 describes the following: In three-dimensional ranging based on the light sectioning method, when scanning and counting of the sensor frame number are started simultaneously, the swing angle of the scanner is determined by knowing the frame in which the slit light is detected, and the distance from the sensor to the object is determined.

[0007] Patent document 3 describes the following: In a shape measuring device that uses a light beam to scan an object to be measured, a light beam spot reflected from a reference surface or the surface of the object to be measured is photoelectrically converted, and the surface shape of the object to be measured is determined based on the time difference between the rising timing of the electrical signal obtained by photoelectric conversion when the reference surface is scanned and the rising timing of the electrical signal obtained by photoelectric conversion when the object to be measured is scanned.

[0008] Patent document 4 describes the following: a three-dimensional image recognition device that irradiates an inspection area on the upper surface of a substrate while conveying a substrate, receives the scanning light reflected by the substrate or an electronic component on the substrate, and calculates the height of the electronic component based on the difference between the light receiving position on the light receiving surface when reflected from the upper surface of the substrate and the light receiving position on the light receiving surface when reflected from the upper surface of the electronic component, wherein the influence of multiple reflections or transmitted reflections is reduced by limiting the incidence of the reflected light on the light receiving surface to only a light-transmitting opening portion set with a predetermined incidence width.

[0009] Patent Document 5 describes a fillet weld sensing device that irradiates a fillet weld between components including a glossy component with slit light, wherein the angle between a light source and a camera is set to an angle at which the image of the slit light is separated from the image of its secondary reflected light.

[0010] Patent Document 6 describes that in a contour shape measuring device based on the light section method, the depth of field is minimized, and images from two or more reflection points are not captured to the same degree as images from a single reflection point, thereby suppressing erroneous measurements caused by multiple reflections.

[0011] Patent document 7 describes the following: In a distance measuring device using a patterned light projection method, an object onto which a line pattern roughly parallel to an epipolar line is projected is photographed, the direction of the line is calculated based on the photographed image, and the multi-reflected light area is detected based on the angular difference between the calculated line direction and the epipolar line direction.

[0012] Patent document 8 describes the following: In an event-based image processing device, whether a predetermined event has occurred in each of a plurality of pixels is detected, an event signal is output based on the detection, timestamp information is generated by mapping at least one pixel corresponding to the event signal with the time of the output event signal, and optical flow is generated based on the timestamp information.

[0013] Patent document 9 describes the following: In a method for three-dimensionally reconstructing a scene, a first continuous event is received for each pixel from a first sensor, a second continuous event is received for each pixel from a second sensor, and a first event in the first continuous event is matched with a second event in the second continuous event based on minimization of a cost function.

[0014] Patent document 10 describes the following: In a three-dimensional measurement method and apparatus for moving a slit light and a camera as a whole along the length direction of an object, obtaining cross-sectional data at each predetermined interval, and performing three-dimensional measurement of the object based on each cross-sectional data, a narrow window is set on an image sensor so as to intersect the slit light perpendicularly, and brightness data with the maximum brightness within the window and the coordinate data in the Y-axis direction at that time are detected.

[0015] Patent Document 11 describes setting an ROI (region of interest) corresponding to an object of interest based on pattern recognition results and tracking the object of interest, thereby suppressing an increase in event data volume and thus a prolonged wait time for event processing.

[0016] Patent Document 12 describes a method for synchronizing an image sensor with structured illumination in a distance measurement system that uses an event detection sensor and structured illumination to measure the distance to a subject, particularly resetting the image sensor in synchronization with switching of patterned light.

[0017] Prior art literature

[0018] Patent Literature

[0019] Patent Document 1: Japanese Patent Application Laid-Open No. 2000-088539

[0020] Patent Document 2: Japanese Patent Application Laid-Open No. 2006-333493

[0021] Patent Document 3: Japanese Patent Application Laid-Open No. 2003-329419

[0022] Patent Document 4: Japanese Patent Application Laid-Open No. 2013-148353

[0023] Patent Document 5: Japanese Patent Application Laid-Open No. 11-33725

[0024] Patent Document 6: Japanese Patent Application Laid-Open No. 2001-227923

[0025] Patent Document 7: Japanese Patent Application Laid-Open No. 2012-141964

[0026] Patent Document 8: Japanese Patent Application Laid-Open No. 2014-002744

[0027] Patent Document 9: Japanese Patent Application Laid-Open No. 2018-516395

[0028] Patent Document 10: Japanese Patent Application Laid-Open No. 7-260444

[0029] Patent Document 11: Japanese Patent Application Publication No. 2020-136958

[0030] Patent Document 12: Japanese Patent Application Publication No. 2021-032763 Summary of the Invention

[0031] Problems to be solved by the invention

[0032] The present invention has been made in view of the conventional problems, and an object of the present invention is to reduce the influence of noise such as multiply reflected reference light when performing distance measurement by optical scanning.

[0033] Means for solving problems

[0034] One embodiment of the present disclosure provides a three-dimensional measurement device comprising: a light projecting unit that projects reference light toward an object while scanning reference light; a light receiving unit that receives the reference light reflected by the object; a time range setting unit that sets a scanning time range for the reference light corresponding to a distance measurement range for each predetermined subarea of ​​the light receiving unit; and a three-dimensional information calculation unit that calculates three-dimensional information of the object by triangulating information from the light receiving unit based on the set scanning time range.

[0035] Another embodiment of the present disclosure provides a three-dimensional measurement device comprising: a light projector configured to project reference light toward an object while scanning the reference light; a light receiver configured to receive the reference light reflected by the object and output only information within a region of interest, and to move the region of interest in accordance with the scanning of the reference light; and a three-dimensional information calculator configured to calculate three-dimensional information of the object by triangulation based on information from the light receiver in the region of interest.

[0036] Effects of the Invention

[0037] According to one embodiment of the present disclosure, reference light directly reflected from a specific portion of an object is guaranteed to be received within the reference light scanning time range corresponding to the ranging range. Therefore, by excluding information from light-receiving portions outside the set scanning time range during ranging, the effects of noise caused by multiple reflections of reference light can be reduced. Furthermore, the ranging range refers to the measurable range predetermined based on the required specifications.

[0038] According to another aspect of the present disclosure, by excluding information from the light receiving unit outside the region of interest during distance measurement, the effects of noise caused by multiply reflected reference light, etc., can be reduced. Furthermore, because the light receiving unit only outputs information from the region of interest, the degradation of the light receiving unit's waiting time due to excessive noise is reduced compared to a case where the light receiving unit's information is limited solely by the scanning time range of the reference light. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a top view of a stereo camera illustrating the measurement principle of the stereo method.

[0040] Figure 2 This is a top view of a light sectioning system illustrating the measurement principle of the light sectioning method.

[0041] Figure 3 This is a block diagram of the three-dimensional measuring device according to the first embodiment.

[0042] Figure 4A This is a top view of the three-dimensional measuring device showing a method for setting a scanning time range.

[0043] Figure 4B This is a plan view of a three-dimensional measuring device showing how the influence of noise such as multiply reflected reference light is reduced.

[0044] Figure 5A It is a plan view of the light-receiving surface showing an example of the set divisions of the scanning time range.

[0045] Figure 5B It is a plan view of the light-receiving surface showing an example of the set divisions of the scanning time range.

[0046] Figure 5C It is a plan view of the light-receiving surface showing an example of the set divisions of the scanning time range.

[0047] Figure 6 This is a flowchart of the three-dimensional measuring device according to the first embodiment.

[0048] Figure 7A This is a plan view showing the three-dimensional measurement device of the region of interest according to the first embodiment.

[0049] Figure 7B This is a plan view showing a three-dimensional measurement device of a region of interest according to a second embodiment.

[0050] Figure 8 This is a block diagram of a three-dimensional measuring device according to a second embodiment.

[0051] Figure 9A This is an image diagram showing an example of setting a region of interest.

[0052] Figure 9B : is an image diagram showing a modified example of setting the region of interest. DETAILED DESCRIPTION

[0053] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In each of the drawings, identical or similar components are denoted by identical or similar reference numerals. In addition, the embodiments described below do not limit the technical scope of the invention described in the scope of protection and the meaning of the terms used.

[0054] The measurement principle of the three-dimensional measuring device of this embodiment will be described. For easier understanding, the measurement principles of the stereo method and the light section method will be described first. Figure 1 This is a top view of a stereo camera 1 illustrating the measurement principle of the stereo method. Stereo camera 1, for example, includes a left light receiving unit 2 and a right light receiving unit 3, which correspond to two cameras. The left light receiving unit 2 and the right light receiving unit 3 are arranged, for example, in parallel and equidistant positions. Specifically, the two light receiving units are separated by a baseline length B, their optical axes are arranged parallel to each other, and the left light receiving surface 4 and the right light receiving surface 5 are arranged in a plane perpendicular to the two optical axes, with the x and y directions of each light receiving surface aligned. Each light receiving surface is, for example, an image sensor comprising a two-dimensional array of multiple pixels, but may also be a line sensor comprising a one-dimensional array of multiple pixels (e.g., arranged only in the x direction).

[0055] Here, the position of the pixel on the left light-receiving surface 4 that captures the image of the point P of the object existing in the target space is defined as x l , set the position of the pixel on the right light-receiving surface 5 to x r When the parallax between the left light receiving part 2 and the right light receiving part 3 is D=x l ﹣x r When the origin of the XYZ coordinate system representing the three-dimensional space is placed at the right focus, the focal lengths of the two light-receiving parts are set to f, and the inter-pixel spacing of the two light-receiving surfaces is set to 1, the distance Z to the point P of the object (the depth to the point P, the same below) is calculated according to the following formula.

[0056] [Formula 1]

[0057]

[0058] The baseline length B and the focal length f are constants determined by the design of the stereo camera 1. Therefore, if the image of point P on the right light-receiving surface 5 corresponding to the image of point P on the left light-receiving surface 4 can be detected through image processing such as pattern matching, the parallax D can be calculated and the distance Z to point P of the object can be calculated.

[0059] The light-cutting system replaces the left light-receiving unit 2 of the stereo camera 1 with a light-projecting unit, for example. Figure 2 This is a top view of a light sectioning system 6 illustrating the measurement principle of the light sectioning method. The light sectioning system 6 includes, for example, a light projector 7. The light projector 7 projects a strip of slit light onto an object while scanning it, and the right light receiving unit 3 receives the slit light reflected from the object. Here, if the projection starting point (rotation center) is located at the left focal point of the stereo camera 1, and the projection angle relative to the left optical axis of the stereo camera 1 is θ, then the pixel position x of the virtual left light receiving surface 4 of the light projector 7 is l It can be calculated by the following formula.

[0060] [Formula 2]

[0061] x f =f tanθ···Formula 2

[0062] In addition, if the light-projecting unit 7 is rotated around the Y-axis perpendicular to the XZ plane at a constant angular velocity ω from the starting point of the light projection to irradiate the strip of slit light, the slit light passes through the left optical axis at time t0 and is projected onto point P of the object at a projection angle θ at time t, then the projection angle θ is calculated by the following formula.

[0063] [Formula 3]

[0064] θ=ω(t-t0) ···Equation 3

[0065] Therefore, at the position x of the pixel on the right light-receiving surface 5, r When the reflected light of the slit light is received at point P, by substituting equations 2 and 3 into equation 1 as shown below, the distance Z to point P of the object can be calculated.

[0066] [Formula 4]

[0067]

[0068] The base length B, the focal length f, the angular velocity ω, and the time t0 are constants determined by the design of the light-cutting system 6. Therefore, if the position x of the pixel on the right light-receiving surface 5 that captures the image of the slit light is calculated, r By calculating the time t at which the image of the slit light is detected, the distance Z to the point P of the object can be obtained.

[0069] The three-dimensional measurement device of this embodiment utilizes the structure and measurement principle of the light section method. However, the above structure and measurement principle are merely examples, and it should be noted that appropriate design changes can be made based on the design of the system structure, layout, and other aspects. For example, the light projector 7 and the right light receiver 3 may be arranged in unequal positions and parallel to each other. Alternatively, instead of replacing the left light receiver 2 with the light projector 7, a system structure may be employed that combines the stereo method with the light section method by providing the light projector 7 in addition to the left and right light receivers 2 and 3. Furthermore, a light projector 7 may be employed that projects a beam of point light or a checkerboard pattern onto the object, rather than projecting a strip of slit light onto the object. It should be noted that such design changes also change the method for calculating three-dimensional information.

[0070] Hereinafter, the configuration of the three-dimensional measuring device according to the first embodiment will be described. Figure 3This is a block diagram of a three-dimensional measuring device 8 according to the first embodiment. The three-dimensional measuring device 8 includes a light projector 7 that projects reference light onto an object W while scanning it, and a light receiver 3 that receives the reference light reflected by the object W. The light projector 7 corresponds to, for example, a projector, and the light receiver 3 corresponds to, for example, a camera. The timing of the light projector 7 and the light receiver 3 are synchronized.

[0071] The light projector 7 projects reference light, such as slit light, spot light, or pattern light, onto the object W. The light projector 7 can project multiple reference light beams at predetermined projection angles. The measurement time of the three-dimensional measuring device 8 is determined by the time it takes to scan the object W with the reference light beams. Therefore, the scanning speed is typically increased to shorten the measurement time. However, this is constrained by the response speed of the light receiving unit 3. Therefore, by projecting multiple reference light beams, the scanning speed can be increased while maintaining the response speed of the light receiving unit 3, shortening the measurement time.

[0072] The light receiving unit 3 has, for example, an image sensor in which a plurality of pixels are arranged two-dimensionally, but may also have a line sensor in which a plurality of pixels are arranged one-dimensionally. The image sensor of the light receiving unit 3 is, for example, an event-based sensor. In the case of an event-based image sensor, the light receiving unit 3 monitors each pixel independently and asynchronously at all times, and when a predetermined event or more (for example, a predetermined brightness change or more) is detected, event information including the position, time, polarity (for example, whether it becomes brighter or darker) of the pixel where the event occurred is output. Alternatively, the image sensor of the light receiving unit 3 may be a general frame-based image sensor. In the case of a frame-based image sensor, the light receiving unit 3 performs exposure by opening and closing the shutter for a predetermined time, thereby outputting a frame image at a predetermined period. The frame image includes, for example, a frame number, brightness information of each pixel, and the like.

[0073] Furthermore, the three-dimensional measurement device 8 includes a three-dimensional information calculation unit 9 that calculates three-dimensional information of the object W through triangulation based on information from the light receiving unit 3. If the sensor of the light receiving unit 3 is an event-based sensor, pixels experiencing an event (e.g., a brightness change greater than a predetermined value) capture an image of the reference light. Therefore, the three-dimensional information calculation unit 9 performs distance measurement based on the event information output from the light receiving unit 3 (e.g., the position, time, and polarity of the pixel experiencing the brightness change). In some cases, the slit width of slit light or the spot diameter of spot light may correspond to multiple pixel sizes. Therefore, distance measurement can be performed by determining the intermediate time between the time when a pixel begins to brighten and the time when it ends to darken. On the other hand, if the sensor of the light receiving unit 3 is a frame-based sensor, the three-dimensional information calculation unit 9 detects the position and frame number (equivalent to the time) of the pixel with the maximum brightness from the multiple frame images output by the light receiving unit 3 and performs distance measurement based on this detection information.

[0074] To mitigate the effects of noise caused by multiple reflections of reference light, the three-dimensional measurement device 8 further includes a time range setting unit 10. This time range setting unit 10 sets a reference light scanning time range corresponding to a distance measurement range for each predetermined section of the light receiving unit 3 (the plurality of pixels constituting the light receiving surface). The three-dimensional information calculation unit 9 excludes information from the light receiving unit 3 outside the set scanning time range and performs distance measurement based on the information from the light receiving unit 3 within the set scanning time range. This reduces erroneous distance measurement caused by noise caused by multiple reflections of reference light, etc.

[0075] The method of setting the scanning time range will be described in detail. Figure 4A This is a top view of a three-dimensional measuring device 8 illustrating the method for setting the scanning time range. The distance measurement range in the figure is the measurable range predetermined based on the required specifications. For example, if the depth of field of the three-dimensional measuring device 8 is designed to enable distance measurement between 1000 mm and 2000 mm, the distance measurement range is 1000 mm to 2000 mm. Since light entering the pixels 11 of the light receiving unit 3 necessarily passes through the line of sight V, if the incident light is a single reflection of the reference light and point P of the object W undergoing the single reflection is within the distance measurement range, it will inevitably be received within the scanning time range of the intersection line segment QR between the reference light scanning line of sight V and the distance measurement range. Therefore, the scanning time range can be set by geometrically calculating the scanning time range of the intersection line segment QR between the reference light scanning pixel 11's line of sight V and the distance measurement range.

[0076] Alternatively, if the distance measurement range is 1000mm to 2000mm, for example, the slit light can be scanned with the flat panel at a distance of 1000mm to record the time when the slit light image passes through each pixel, and then scanned with the flat panel at a distance of 2000mm to record the time when the slit light image passes through each pixel. These two times are used as the maximum and minimum values ​​of the scanning time range. Because measurement variation may occur, it is practical to perform multiple measurements and average them, or to set the scanning time range with a slight margin. By performing distance measurement based on information from the light receiving unit 3 within this set scanning time range, erroneous distance measurements caused by noise caused by multiple reflections of reference light can be suppressed.

[0077] Figure 4B This is a top view of three-dimensional measuring device 8 showing how the effects of multiply reflected reference light are mitigated. When reference light, after being reflected once by a surface S of an object W, is reflected twice at a point P of the object W and enters a pixel 11, it is more likely to enter that pixel 11 outside the set scanning time range. Therefore, by performing distance measurement while excluding information from the light receiving unit 3 outside the set scanning time range, erroneous distance measurements caused by multiply reflected reference light can be suppressed.

[0078] Regarding the scanning time range, for each of the multiple pixels constituting the light receiving part 3, one pixel among the multiple pixels can be set as a partition, but for the purpose of saving setting memory and shortening measurement time, a group of adjacent pixels among the multiple pixels constituting the light receiving part 3 can be aggregated and set as a partition for each partition. Figures 5A to 5C FIG. 5 is a top view of the light receiving surface 5 showing an example of a set division of the scanning time range. Assuming that the assembly error of the three-dimensional measuring device 8, lens deformation, etc. can be ignored, as shown in FIG. Figure 5A As shown in FIG, when a slit light is used to scan a flat plate in the x direction, and the image I of the slit light is formed parallel to the y axis of the light receiving surface 5, the pixel group of one or more columns that simultaneously captures the image I of the slit light can be grouped into one partition to set the scanning time range (for example, 0.2s to 0.4s). Figure 5B As shown, when a checkerboard-shaped pattern light is used to scan a flat plate in the x-direction, when the image I of the pattern light is imaged parallel to the y-axis of the light-receiving surface 5, two or more columns of pixel groups that simultaneously capture the image I of the pattern light can be aggregated into a partition to set the scanning time range (for example, 0.15s to 0.4s).

[0079] And, as Figure 5C As shown, when a flat panel is scanned in the x and y directions using a spot light, and an image I of the spot light is formed in four adjacent pixel groups on the light-receiving surface 5, the four adjacent pixel groups that simultaneously capture the image I of the spot light can be grouped into a single segment to set a scanning time range (e.g., 0.2s to 0.4s). The scanning time range thus set can be the range of scanning times elapsed since the start of scanning in the case of an event-based image sensor, or the range of frame numbers elapsed since the start of scanning in the case of a frame-based image sensor.

[0080] Refer again Figure 3 In the case where the light projecting unit 7 projects a plurality of slit lights, the three-dimensional measuring device 8 may include a projection angle interval setting unit 12 for setting the projection angle interval according to the set scanning time range. The projection angle interval is set to an angle interval wider than the scanning angle φ, for example. Figure 4A and Figure 4BThe scanning angle of the slit light scan within the scanning time range set as shown. If a plurality of slit lights are projected while maintaining a projection angle interval narrower than the scanning angle φ, each pixel of the partition for which the scanning time range of the scanning angle φ is set receives a plurality of slit lights within the scanning time range, and therefore a method is required to identify which slit light is received. However, by setting the projection angle interval of the plurality of slit lights to an angle interval wider than the scanning angle φ, each pixel in the partition receives only one slit light within its scanning time range, and therefore there is no need to identify the slit light. When projecting a plurality of Figure 5B In the case of the checkerboard pattern light as shown, the projection angle interval can be set in the same manner.

[0081] Furthermore, regardless of whether the light receiving unit 3 comprises an event-based image sensor or a frame-based image sensor, the image sensor of the light receiving unit 3 only needs to be able to output information within a region of interest. The region of interest can be a well-known region of interest (ROI), but it should be noted that the term "region of interest" in this specification is not necessarily limited to a general ROI. For example, a frame-based image sensor outputs only the captured image within the region of interest, while an event-based image sensor outputs only information about events occurring within the region of interest. If the light receiving unit 3 is able to output only information within the region of interest, the three-dimensional measurement device 8 may include a region of interest moving unit 14 that moves the region of interest based on scanning of reference light. The light receiving unit 3 moves the region of interest based on an input signal (e.g., a movement command) from the region of interest moving unit 14. For example, the region of interest moves over time to encompass a segment (pixel or pixel group) for which a scanning time range is set, where the scanning time range includes a certain time t during measurement. The segment (pixel or pixel group) for which the scanning time range is set does not necessarily need to completely coincide with the region of interest. This prevents the region of interest from missing event occurrences or outputting maximum brightness. This region of interest limits the pixels where events occur and pixels that output maximum brightness, thereby reducing the amount of data output from the light receiving unit 3. The three-dimensional information calculation unit 9 then performs distance measurement based on the event information restricted within the region of interest, or detects the position and frame number of the pixel with maximum brightness within the restricted region of interest for distance measurement, significantly shortening measurement time. Furthermore, since noise caused by multiple reflections of reference light is also limited by the region of interest, erroneous distance measurement caused by noise can be suppressed.

[0082] Refer again Figure 3The three-dimensional information calculated by the three-dimensional information calculation unit 9 is output to the outside of the three-dimensional measurement device 8. The three-dimensional information is used not only by a so-called image processing device but also by an external device 13 such as a robot control device or a vehicle control device. The external device 13 can perform image processing based on the three-dimensional information, which reduces the influence of noise caused by multiply reflected reference light, and can perform position control, velocity control, acceleration control, and the like, for example.

[0083] The above-mentioned three-dimensional information calculation unit 9, time range setting unit 10, projection angle interval setting unit 12 and region of interest moving unit 14 can be installed, for example, through an integrated circuit such as ASIC (application specific integrated circuit) or FPGA (field-programmable gate array), or can also be installed as a program executed by a processor such as CPU (central processing unit) or MPU (micro processing unit).

[0084] Next, the operation of the three-dimensional measuring device 8 will be described. Figure 6 This is a flowchart illustrating the operation of the three-dimensional measuring device 8. In step S1, the time range setting unit 10 sets a reference light scanning time range corresponding to the distance measurement range for each predetermined sub-area of ​​the multiple pixels constituting the light receiving unit 3. If the reference light scanning time range is geometrically calculated, the user can input the distance measurement range via a user interface (not shown), such as a touch panel display or keyboard, or a distance measurement range pre-stored in memory can be used. If multiple reference lights are projected, in step S2, the projection angle interval setting unit 12 sets the projection angle interval based on the set scanning time range.

[0085] Once the projection and scanning of the reference light begins in step S3, in step S4, the three-dimensional information calculation unit 9 calculates the three-dimensional information of the object by triangulating the information from the light receiving unit 3 within the set scanning time range. If the image sensor of the light receiving unit 3 is an event-based image sensor, the three-dimensional information of the object is calculated based on the position, time, and polarity of pixels experiencing events within the set scanning time range. If the image sensor of the light receiving unit 3 is a frame-based image sensor, the position and frame number of the pixel with the maximum brightness are detected from multiple frame images within the set frame number range, and the three-dimensional information of the object is calculated based on the detected pixel position and frame number. The processing ends when the three-dimensional information is output to the outside in step S5.

[0086] In this flowchart, steps S1 to S5 are described as a series of processes. However, steps S1 and S2 can also be performed during setup or calibration of the three-dimensional measuring device, and steps S3 to S5 can also be performed during measurement. It should be noted that steps S2 and S5 are not necessarily required.

[0087] According to the three-dimensional measuring device 8 of the first embodiment, the reference light directly reflected from a specific portion of the object W is necessarily received within the scanning time range corresponding to the ranging range. Therefore, by excluding information from the light receiving unit 3 outside the set scanning time range during ranging, the influence of noise such as multiply reflected reference light can be reduced.

[0088] Hereinafter, a three-dimensional measuring device 8 according to a second embodiment will be described. Note that descriptions of the same configurations and operations as those of the three-dimensional measuring device 8 according to the first embodiment will be omitted. Figure 7A FIG. 1 is a plan view showing the three-dimensional measurement device 8 of the region of interest according to the first embodiment. Figure 7B This is a top view of a three-dimensional measuring device 8 showing a region of interest according to the second embodiment. In the three-dimensional measuring device 8 of either the first or second embodiment, while reference light L scans the projection angle range θmax of the entire distance measurement range at a scanning speed ω, the region of interest moves in a movement direction d in response to (following) the scanning of reference light L. However, the three-dimensional measuring devices 8 of the first and second embodiments differ in the following respects.

[0089] like Figure 7A As shown, in the three-dimensional measurement device 8 of the first embodiment, focusing on one pixel 11, under the assumption that the reference light L imaged by the pixel 11 should be in the scanning time range of the reference light L between the time t1 at which the reference light L is imaged by the pixel 11 and the time t2 at which the reference light L is imaged and the point Q on the closest side of the ranging range is illuminated (i.e., the scanning time range of the reference light L corresponding to the ranging range), the region of interest moves over time so as to include a partition for which the following scanning time range is set, wherein the scanning time range includes a certain time in the measurement.

[0090] In contrast, Figure 7BAs shown, in the three-dimensional measuring device 8 of the second embodiment, focusing on a time t, it is presumed that the image of the reference light L irradiated at a certain time t during measurement should be within the imaging range between the imaging point when reflected at the point Q on the nearest side of the ranging range (for example, pixel 11) and the imaging point when reflected at the point R on the farthest side of the ranging range (for example, pixel 15) (i.e., the imaging range of the reference light L corresponding to the ranging range). The region of interest moves over time so that the imaging range of the reference light L corresponding to the ranging range (for example, the pixel group from pixel 11 to pixel 15) is included at a certain time t during measurement.

[0091] Figure 8 This is a block diagram of a three-dimensional measurement device 8 according to a second embodiment. In the three-dimensional measurement device 8 of either the first or second embodiment, the region of interest (ROI) moves in synchronization with the scanning of reference light. However, in the first embodiment, the ROI moving unit 14 sends a movement command to the light receiving unit 3 to move the ROI so that the region (e.g., pixel 11 or a group of pixels near pixel 11) includes a segment (e.g., pixel 11 or a group of pixels near pixel 11) for which the scanning time range of reference light L corresponding to the distance measurement range is set. In contrast, in the second embodiment, the three-dimensional measurement device 8 includes a ROI setting unit 16 that pre-sets the ROI. The ROI setting unit 16 sets the ROI so that the imaging range of the reference light corresponding to the distance measurement range is included at a certain time t during measurement. The light receiving unit 3 itself moves the set ROI based on an input signal (e.g., a trigger signal) from the light projecting unit 7. The ROI setting unit 16 sets at least one of the initial position, size (width and height), movement direction, and movement speed of the ROI. The light receiving unit 3 then places the ROI of the set size at the initial position and moves it from the initial position in a predetermined direction at a predetermined speed (e.g., a constant speed). The light receiving unit 3 can move the region of interest by receiving an input signal (e.g., a trigger signal) when the projection angle of the reference light L reaches a predetermined angle. Furthermore, the light receiving unit 3 can move the region of interest in synchronization with a clock signal from the image sensor (light receiving surface 5).

[0092] Figure 9A This is an image diagram showing an example of setting a region of interest. The region of interest's initial position p0, dimensions (width w and height h), movement direction d, and movement speed v are set based on the design information of the three-dimensional measuring device 8. The design information of the three-dimensional measuring device 8 includes, for example, the configuration of the light projector 7 and light receiver 3 (e.g., their respective positions and postures), the distance measurement range, the scanning direction of the reference light L, the scanning speed of the reference light L, the projection angle range of the reference light L, and the clock frequency of the image sensor.

[0093] When setting the region of interest, the region of interest setting unit 16 first calculates the imaging range i of the reference light L. The region of interest setting unit 16 may have a function of inputting design information of the three-dimensional measurement device 8 (the configuration of the light projecting unit 7 and the light receiving unit 3 (their respective positions and postures), the distance measurement range, etc.) via a user interface unit (not shown) such as a touch panel display or a mouse, and geometrically calculating the imaging range i of the reference light L based on the design information. Alternatively, the region of interest setting unit 16 may have a function of actually scanning the reference light L to calculate the imaging range i of the reference light L. In the latter case, for example, the reference light L is scanned. Figure 7B To illustrate, a flat plate is placed at the closest side of the ranging range, reference light L is scanned, and the position and time of the image point (e.g., pixel 11) formed by the reference light L are recorded. Separately, a flat plate is placed at the farthest side of the ranging range, reference light L is scanned, and the position and time of the image point (e.g., pixel 15) formed by the reference light L are recorded. The image range i of the reference light L corresponding to the ranging range increases toward the end of the light-receiving surface 5 and becomes constant due to the scanning of the reference light L. Therefore, the region of interest setting unit 16 can calculate the number of pixels from the image point (e.g., pixel 11) formed by the reference light L at the closest side of the ranging range to the image point (e.g., pixel 15) formed by the reference light L at the farthest side of the ranging range at each moment, and calculate the maximum value of this number of pixels across the entire ranging range as the image range i of the reference light L.

[0094] The region of interest setting unit 16 sets the width w of the region of interest based on the imaging range i of the reference light L. The width w of the region of interest refers to the size of the region of interest in the scanning direction of the reference light L. The initial value of the width w of the region of interest can be the imaging range i of the reference light L itself (for example, the maximum number of pixels from pixel 11 to pixel 15 that have passed through the entire ranging range). However, even if the scanning speed of the reference light L is a constant angular velocity, the scanning speed of the reference light L formed on the light-receiving surface 5 is not constant. Therefore, a margin is sufficient for the width w of the region of interest. For example, the region of interest setting unit 16 can set the width w of the region of interest as the size obtained by adding a predetermined margin m to the imaging range i of the reference light L. The margin m for the width of the region of interest can be a predetermined constant.

[0095] The region of interest setting unit 16 sets the height h of the region of interest based on the size of the light-receiving surface 5. The height h of the region of interest refers to the size of the region of interest in a direction perpendicular to the scanning direction of the reference light L. The initial value of the height h of the region of interest can be the number of pixels on the light-receiving surface 5 perpendicular to the scanning direction of the reference light L. However, the region of interest setting unit 16 can also specify and set the height h of the region of interest using a user interface or the like, using an arbitrary value less than the number of pixels on the light-receiving surface 5 perpendicular to the scanning direction of the reference light L. For example, when measuring the three-dimensional shape of an object reflected in a known area of ​​the light-receiving surface 5, the height h of the region of interest can be limited to the area where the object exists, thereby reducing the influence of noise.

[0096] The region of interest setting unit 16 sets the moving direction d of the region of interest based on the scanning direction of the reference light L. The region of interest setting unit 16 can arbitrarily designate and set the moving direction d of the region of interest via a user interface or the like, or can determine the scanning direction of the reference light L when executing the above-mentioned function of scanning the reference light L to calculate the imaging range i of the reference light L, and automatically set the scanning direction of the reference light L as the moving direction d of the region of interest. Figure 9A In the example shown in FIG5 , the reference light (slit light) is arranged to be projected horizontally onto the light-receiving surface 5, and the moving direction d of the region of interest is set to the upward direction or the downward direction. On the other hand, when the reference light (slit light) is arranged to be projected vertically onto the light-receiving surface 5, the moving direction d of the region of interest is set to the left direction or the right direction.

[0097] The region of interest setting unit 16 sets the initial position p0 of the region of interest using the coordinate values ​​of an orthogonal coordinate system (e.g., a camera coordinate system (xy coordinate system shown in the figure)) fixed on the same plane as the light receiving surface 5. For example, when the region of interest is rectangular, the initial position p0 of the region of interest may be the coordinate value of the upper left corner of the rectangle. The region of interest setting unit 16 may also automatically set the initial position p0 of the region of interest based on the scanning direction of the reference light L (or the moving direction d of the region of interest) and the size (width and height) of the region of interest. For example, in Figure 9AIn the example shown in FIG. 5 , the reference light L (slit light) is configured to be projected horizontally onto the light-receiving surface 5 and scanned upward or downward onto the light-receiving surface 5. Therefore, the region of interest setting unit 16 automatically sets the position of the upper left corner of the region of interest, with the bottommost or topmost pixel row on the light-receiving surface 5 serving as the center line, as the initial position p0 of the region of interest. Alternatively, if the reference light L (slit light) is configured to be projected vertically onto the light-receiving surface 5 and scanned rightward or leftward onto the light-receiving surface 5, the position of the upper left corner of the region of interest, with the leftmost or rightmost pixel row on the light-receiving surface 5 serving as the center line, may be set as the initial position p0 of the region of interest. Alternatively, the region of interest setting unit 16 may arbitrarily designate and set the initial position p0 of the region of interest via a user interface or the like. For example, the reference light L (slit light) is arranged to be projected transversely relative to the light-receiving surface 5, and the reference light L is scanned downwardly from the light-receiving surface 5. When measuring the three-dimensional shape of an object projected in a known area of ​​the light-receiving surface 5, the initial position p0 of the region of interest is automatically set to the position of the upper left corner of the region of interest, with the center line being the topmost pixel row in the area where the workpiece is located. This allows the scanning of the region of interest to be limited to the area where the object is located, thereby reducing the influence of noise.

[0098] The region of interest setting unit 16 sets the movement speed v of the region of interest based on the scanning speed ω of the reference light L. When the region of interest is moved synchronously with the image sensor's clock signal, the region of interest setting unit 16 determines, based on the movement speed v, how many clocks of the image sensor's operating clock correspond to the time it takes for the reference light L to move one pixel on the light-receiving surface 5. The light-receiving unit 3 then moves the region of interest by one pixel for each clock cycle required for the reference light L to move one pixel on the light-receiving surface 5. For example, the time t required for the reference light L to scan the entire ranging range is calculated as projection angle range / scanning speed (t = θmax / ω). Therefore, if the number of pixels on the light-receiving surface 5 in the scanning direction is n, the time Δt required for the image of the reference light L to move one pixel on the light-receiving surface 5 is Δt ≈ t / n. In other words, if the image sensor's clock frequency is F, the number of clock cycles c required for the reference light L to move one pixel on the light-receiving surface 5 is c = F / v. The light receiving unit 3 then moves the region of interest by one pixel for each clock count c required for the reference light L to move one pixel on the light receiving surface 5. By moving the region of interest in synchronization with the clock signal of the image sensor, the region of interest can be moved with a simpler configuration than when the light receiving unit 3 receives an external signal as a trigger signal.

[0099] Based on an input signal (e.g., a trigger signal) from the light projector 7, the light receiving unit 3 places a region of interest (ROI) of a set size (width w and height h) at an initial position p0 and moves it in a direction d at a speed v. Over time, the ROI moves from the initial position p0 to position p1. The three-dimensional information calculation unit 10 calculates three-dimensional information about the object by triangulating information from the light receiving unit 3 (event information or luminance image) based on the ROI. By excluding information from the light receiving unit 3 outside the ROI during distance measurement, the three-dimensional information calculation unit 10 can reduce the effects of noise caused by multiple reflections of reference light, etc.

[0100] Figure 9B : is an image diagram showing a modified example of setting the region of interest. When the light projecting unit 7 projects a plurality of reference lights L1 and L2 while maintaining a predetermined projection angle interval, the region of interest setting unit 16 individually sets the initial positions p of the plurality of regions of interest 1 and 2 based on the projection angle interval. 10 、p 20 The initial positions p of the multiple regions of interest 1 and 2 are 10 、p 20 The other settings (size (width w and height h), moving direction d, moving speed v, etc.) can be the same. When the projection angles of the reference lights L1 and L2 reach the predetermined angles, the light receiving unit 3 receives an input signal (e.g., a trigger signal) from the light projecting unit 7, and places the regions of interest 1 and 2 of the set size (width w and height h) at the initial position p, respectively. 10 、p 20 , and moves it in the direction of movement d at a moving speed v. Furthermore, the light receiving unit 3 is not limited and may simply output information (event information or a luminance image) representing the logical AND region of the region of interest and the sub-area for which the reference light scanning time range is set, as described in the first embodiment. The three-dimensional information calculation unit 9 calculates three-dimensional information of the object through triangulation based on this information.

[0101] Although not shown, the region of interest setting unit 16 may also have a function of setting a provider-side region of interest (ROI) set on the provider side of the three-dimensional measuring device 8 and a user-side region of interest (ROI) set on the user side of the three-dimensional measuring device 8. The region of interest setting unit 16 designates and sets the provider-side ROI and the user-side ROI via a user interface, etc. In this case, in addition to the aforementioned partitions for which the reference light scanning time range is set, the region of interest setting unit 16 may also set a region as a logical AND of the provider-side ROI and the user-side ROI.

[0102] According to the second embodiment of the three-dimensional measurement device 8, by excluding information from the light receiving unit 3 outside the region of interest during distance measurement, the effects of noise caused by multiple reflections of reference light, etc., can be reduced. Furthermore, since the light receiving unit 3 (i.e., the image sensor) only outputs information (event information or brightness information) from the region of interest, the degradation of image sensor processing time due to excessive noise is reduced compared to a case where the information output from the image sensor is limited solely based on the scanning time range of the reference light. In particular, since event-based image sensors only output event information, latency is inherently low. By outputting only event information from the region of interest, degradation of image sensor latency (delay time) due to excessive events can be mitigated, allowing the advantages of event-based image sensors to be realized or maintained.

[0103] In addition, the above-mentioned program executed by the processor, integrated circuit, etc. can be recorded on a computer-readable non-transitory recording medium, such as a CD-ROM, or can be distributed and provided via wired or wireless means from a server device on a WAN (wide area network) or LAN (local area network).

[0104] Although various embodiments have been described in this specification, the present invention is not limited to the above-described embodiments, and it should be understood that various modifications can be made within the scope described in the scope of protection.

[0105] Description of Reference Signs

[0106] 1 Stereo Camera

[0107] 2Left light receiving part

[0108] 3Right light receiving part (light receiving part)

[0109] 4 Left light-receiving surface

[0110] 5Right light-receiving surface (light-receiving surface)

[0111] 6-light cutting system

[0112] 7Light projection department

[0113] 8. Three-dimensional measuring device

[0114] 9Three-dimensional Information Computing Department

[0115] 10 Time range setting unit

[0116] 11 pixels

[0117] 12 Projection angle interval setting unit

[0118] 13 External devices

[0119] 14 Concern for Regional Mobility

[0120] B baseline length

[0121] D-parallax

[0122] f focal length

[0123] Reference light image

[0124] L reference light

[0125] P object point

[0126] QThe imaging point at the nearest side of the ranging range

[0127] R The imaging point at the farthest side of the ranging range

[0128] The intersection of the QR pixel's line of sight and the ranging range

[0129] S The surface of the object

[0130] V sight

[0131] W object

[0132] The distance from Z to point P of the object

[0133] θ projection angle

[0134] θmax projection angle range

[0135] φ scanning angle

[0136] ω scanning speed

[0137] iImaging range of reference light

[0138] p0, p 10 、p 20 Initial position of the region of interest

[0139] wThe width of the region of interest

[0140] m is the width margin of the region of interest

[0141] hHeight of the area of ​​interest

[0142] d The moving direction of the area of ​​interest

[0143] v Moving speed of the area of ​​interest

[0144] F is the clock frequency of the image sensor.

Claims

1. A three-dimensional measuring device, characterized in that: have: a light projecting unit configured to project the reference light toward the object while scanning the reference light; a light receiving unit that receives the reference light reflected by the object; a time range setting unit that sets, for each predetermined subarea of ​​the light receiving unit, a scanning time range of the reference light between a time when the farthest point in the distance measuring range is irradiated and a time when the closest point in the distance measuring range is irradiated; and a three-dimensional information calculation unit that calculates three-dimensional information of the object by performing triangulation based on information of the light receiving unit only within the set scanning time range; The light receiving unit includes a plurality of pixels, monitors an event for each pixel, and outputs at least a position, a time, and a polarity of the pixel where the event occurs as the information.

2. The three-dimensional measuring device according to claim 1, wherein: The light projecting unit projects the plurality of reference lights while maintaining a predetermined projection angle interval.

3. The three-dimensional measuring device according to claim 1 or 2, characterized in that: The scanning timing range is a scanning timing range of an intersection line segment between a line of sight in which the reference light scans the pixels of the light receiving unit and the distance measurement range.

4. The three-dimensional measuring device according to claim 1 or 2, characterized in that: The scanning time range is set for each pixel by treating one pixel in the light receiving section as a sub-area, or the scanning time range is set for each sub-area by treating a group of adjacent pixels in the light receiving section as a sub-area.

5. The three-dimensional measuring device according to claim 2, wherein: The three-dimensional measuring device further includes a projection angle interval setting unit configured to set the projection angle interval based on the set scanning time range.

6. The three-dimensional measuring device according to claim 1 or 2, characterized in that: The light receiving unit has a function of outputting the information only within a region of interest among the plurality of pixels, the region of interest moving with the passage of time to include the partition set with a scanning time range including a certain time during measurement.

7. The three-dimensional measuring device according to claim 1 or 2, characterized in that: The light receiving unit has a function of outputting only the information within a region of interest among a plurality of pixels, and the region of interest moves with time so as to include an imaging range of the reference light corresponding to the distance measurement range at a certain point in time during measurement.

Images