Depth data measuring device and its application method

Abstract

A depth data measurement device and its application method are disclosed. The measurement device includes a projection assembly for projecting a line of light onto an imaging target in the presence of relative motion; an image sensor for performing imaging based on reflected light from the line of light, wherein each pixel of the image sensor transmits a valid signal with a timestamp when the change in the amount of light received within a unit time exceeds a threshold; and a processor for calculating surface depth information of the imaging target based on the pixel position corresponding to the valid signal, the timestamp of the valid signal, and relative motion information. The present invention combines active illumination and imaging of an object in the presence of relative motion using a line of light, an image sensor in which each pixel can independently generate a valid signal when its brightness changes, imaging based only on the valid signal, and determining the position of depth data using the timestamp and relative motion information, thereby realizing high-speed acquisition and processing of depth information with a simple projection mechanism and extremely low computational power requirements.

Description

[Technical Field]

[0001] The present disclosure relates to the field of three-dimensional imaging, and in particular to a depth data measurement device and its application method. [Background technology]

[0002] A depth camera is a collection device that collects depth information of a target object, and such cameras are widely applied in fields such as 3D scanning and 3D modeling. For example, nowadays, more and more smartphones are equipped with depth cameras for face recognition.

[0003] The prior art tends to use increasingly complex devices and projection patterns to achieve high precision imaging, which generally requires high frame rates, which places very high demands on the computing power and bandwidth of the processor of the depth data measurement device.

[0004] Therefore, there is a need for a means to obtain more accurate depth information at a lower computational cost. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem to be solved by the present disclosure is to provide a depth data measurement device and its application method, which uses a line light to actively illuminate and image an object in the presence of relative motion, combines with an image sensor in which each pixel can independently generate an effective signal when the brightness changes, and can perform imaging only on the effective signal, and uses timestamps and relative motion information to determine the position of the depth data, thereby realizing high-speed acquisition and processing of depth information with a simple projection mechanism and extremely low computational power requirements. [Means for solving the problem]

[0006] According to a first aspect of the present disclosure, there is provided a depth data measurement device including: a projection assembly for projecting a line of light onto an imaging object in the presence of relative motion; an image sensor for imaging the imaging object based on reflected light of the line of light, the image sensor including a plurality of pixels, each of which transmits a valid signal accompanied by a timestamp when a change in the amount of light received within a unit time exceeds a threshold; and a processor for calculating surface depth information of the imaging object based on the position of the pixel corresponding to the valid signal, the timestamp of the valid signal, and relative motion information.

[0007] Alternatively, the relative motion is a motion of the imaging object relative to the depth data measuring device, and when the depth data measuring device is stationary, motion information of the imaging object is acquired as the relative motion information, when the imaging object is stationary, motion information of the depth data measuring device is acquired as the relative motion information, or when both the imaging object and the depth data measuring device are in a moving state, relative motion information of the imaging object and the depth data measuring device is acquired as the relative motion information.

[0008] Optionally, the depth data measuring device further includes a motion information acquiring device used to acquire the relative motion information.

[0009] Optionally, the image sensor includes a first image sensor and a second image sensor installed on either side of the projection assembly, and the processor calculates depth information of the imaging object using positions of pixels corresponding to valid signals from the first image sensor and the second image sensor, timestamps of the valid signals, and the first relative motion velocity.

[0010] Optionally, the projection assembly projects line light pulses at a predetermined frequency.

[0011] Optionally, the relative movement is movement of a line of light projected by the projection assembly relative to the image sensor and the depth data measurement device, and the projection assembly performs scanning projection with the line of light by rotation of a rotation mechanism.

[0012] Optionally, the projection assembly scan-projects a line of light moving along a first direction onto the imaging area by rotating the rotation mechanism, wherein the length direction of the line of light is a second direction perpendicular to the first direction, and the scan-projected line of light forms a striped light pattern by changing brightness, and the processor is used to generate a two-dimensional image of the striped light pattern based on the brightness of the striped light pattern corresponding to the pixel position and the timestamp corresponding to the valid signal, and to derive surface depth information of the imaging object in the imaging area based on the two-dimensional image.

[0013] Alternatively, the projection assembly completes one pattern scan projection within a scan period T, and within the scan period T, the projection assembly adjusts the brightness of the line light in the moving projection based on one stripe light pattern to be projected, and the projection assembly adjusts the brightness of the line light based on different stripe light patterns within N consecutive scan periods T, and the processor generates N two-dimensional images corresponding to the N stripe light patterns based on the N scan periods T, and synthesizes one depth image of the imaging object in the imaging area based on the N two-dimensional images.

[0014] Optionally, the projection assembly completes one pattern scan within a scan period T, and within the scan period T, the projection assembly alternately adjusts the brightness of the line light in the moving projection based on the N stripe light patterns to be projected, and the processor generates N two-dimensional images corresponding to the N stripe light patterns based on the one scan period T, and synthesizes one depth image of the imaging object in the imaging area based on the N two-dimensional images.

[0015] Optionally, the projection assembly completes one scan within a scan period T, and the processor assigns a brightness corresponding to a pre-specified stripe light pattern to the pixel location corresponding to the timestamp.

[0016] According to a second aspect of the present disclosure, there is provided a method for measuring volume information of an item on a conveyor belt, which is executed by a depth data measurement device according to the first aspect, the method including the steps of projecting a line of light toward an item on a conveyor belt, imaging a projected area of the item based on reflected light of the line of light and obtaining an effective signal accompanied by a timestamp, calculating depth information of the item corresponding to the timestamp based on the pixel position corresponding to the effective signal, obtaining the movement information of the conveyor belt as the movement information of the item, and calculating the volume information of the item based on the depth information of the item at multiple timestamps and the movement information of the item.

[0017] Optionally, the conveyor belt has a first direction in which it moves, and the length of the line of light is a second direction perpendicular to the first direction. Optionally, the projection direction of the line of light is perpendicular to the conveyor belt surface or a tangent to the conveyor belt surface.

[0018] According to a third aspect of the present disclosure, there is provided a method for detecting the surface state of an object, which is executed by a depth data measuring device according to the first aspect, and the detection method includes the steps of projecting a line of light toward the surface of the object, imaging the surface of the object based on reflected light of the line of light and acquiring an effective signal accompanied by a timestamp, calculating depth information of the surface of the object corresponding to the timestamp based on the pixel position corresponding to the effective signal, acquiring motion information of an exercise equipment to which the depth data measuring device is attached, and determining the state of the surface position of the object corresponding to each timestamp based on the depth information and the motion information of the surface of the object at multiple timestamps.

[0019] Optionally, the surface condition of the object can be determined by comparing depth information of the surface of the object at multiple timestamps with depth information of the surface positions of the object corresponding to the predetermined depth distribution pattern, and / or an actual surface depth distribution of the object is synthesized based on the depth information of the surface of the object at multiple timestamps and the surface positions of the object corresponding to each timestamp, and the road surface condition is determined based on the road surface depth distribution of the object surface.

[0020] According to a fourth aspect of the present disclosure, there is provided a method for measuring depth data, including the steps of projecting a line of light onto an imaging object in which relative motion exists, and relative motion exists between the line of light and the imaging object; imaging the imaging object based on reflected light of the line of light, and each pixel transmitting a valid signal with a timestamp when the change in the amount of light it receives within a unit time exceeds a threshold; and calculating the surface depth information of the imaging object based on the position of the pixel corresponding to the valid signal, the timestamp of the valid signal, and the relative motion information.

[0021] Optionally, the step of projecting the line light onto the imaging object further includes the steps of scanningly projecting the line light onto the imaging object by rotating a rotation mechanism, and projecting the line light in a fixed direction relative to a housing of the depth data measuring device, wherein there is relative motion between the depth data measuring device and the imaging object. [Brief explanation of the drawings]

[0022] JPEG2025525974000002.jpg133170 DETAILED DESCRIPTION OF THE INVENTION

[0023] Preferred embodiments of the present disclosure will be described in more detail below with reference to the drawings. While the drawings illustrate preferred embodiments of the present disclosure, it should be understood that the present disclosure can be embodied in various forms and should not be limited to the embodiments set forth herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0024] In the field of computer vision technology, which derives depth information (three-dimensional information) of an object by actively projecting light and imaging it based on the object's reflection of the projected light, structured light having a specific distribution pattern in two dimensions is generally selected.

[0025] Structured light technology is a technology that actively projects a known pattern onto an imaging area, and information about the distortion of the structured light pattern that occurs on the surface of the imaging object in the imaging area can be used in a vision system to calculate surface depth information of the imaging object. The structured light needs to be coded, for example, a widely used structuring is stripe code structured light.

[0026] Figures 1A-B show an example of projecting stripe-structured light using a time code. The diagram briefly explains the encoding principle of stripe-structured light using a 5-bit binary time code with two grayscales. The projection assembly can sequentially project five patterns, as shown in Figure 1A, onto the imaging area. Each of these five patterns uses two grayscales, light and dark, to divide the projection space into 32 regions. The first projected pattern contains only two regions: dark on the left and light on the right. The second pattern contains four regions, dark-light-dark-light, from left to right, and so on. The fifth pattern contains 32 regions formed by alternating dark and light regions.

[0027] In each pattern, it can be assumed that each region corresponds to a different projection angle, with bright regions corresponding to the code "1" and dark regions corresponding to the code "0." The code values in the five encoding patterns of a point on an object in the projection space are combined according to the projection order to obtain the region code value of the point, thereby identifying the region in which the point is located and decoding the scan angle of the point. FIG. 1B is a schematic diagram of encoding based on five patterns. As shown in the figure, each pixel receives a code value of "1" or "0" in each of the five images, thereby obtaining a 5-bit binary code value. In a monocular imaging system (i.e., only one image sensor images the five patterns), the decoding process may include comparison with and calculation of reference image code values. In a binocular imaging system, the decoding process can be simplified by directly matching the code values of each point in the first and second image sensors.

[0028] To obtain the five patterns shown in Figure 1A by flashing during the scan period, assuming that the time to project one pattern is 32 ms, in the first 32 ms, the line light is first turned off for 16 ms and then turned on for 16 ms as the scan position changes. In the second 32 ms, the line light is first turned off for 8 ms, then turned on for 8 ms, then turned off for 8 ms, then turned on for 8 ms. In the third 32 ms, the line light is first turned off for 4 ms and then turned on for 4 ms as the scan position changes. This is repeated four times. In the fourth 32 ms, the line light is first turned off for 2 ms and then turned on for 2 ms as the scan position changes, and this is repeated eight times. In the fifth 32 ms, the line light is first turned off for 1 ms and then turned on for 1 ms as the scan position changes, and this is repeated 16 times. This results in five scan patterns in five 32 ms.

[0029] To improve matching accuracy, the number of projection patterns in the time code can be increased to, for example, six, seven, or more. For example, in a twin-eye imaging application, a stripe light pattern with ten time codes means that each pixel in each left and right image frame contains ten domain code values of 0 or 1, thereby achieving higher accuracy (e.g., pixel level) for left and right image matching. When the projection speed of the projection assembly remains unchanged, the ten-code pattern example corresponds to achieving higher accuracy for image matching at a higher time domain cost (double the time) compared to the five-code pattern in FIG. 1A.

[0030] In order to project the stripe code structured light, the reflecting mechanism needs to perform scanning projection with respect to the line light within a predetermined round-trip angle, and form corresponding multiple stripe light patterns by combining corresponding flashing modes of the line light, which is complex in form and requires a large amount of computing power. In view of this, the present disclosure provides a depth data measurement means in an application scene where there is relative motion between the projected line light and the imaging object, by simply projecting a line light and combining it with an image sensor in which each pixel can independently generate an effective signal when the brightness changes, which provides an extremely fast depth information acquisition speed and extremely low computing power requirements.

[0031] 2 is a schematic diagram of a depth data measurement device according to one embodiment of the present disclosure. The left side of the figure shows how the depth data measurement device 200 projects a line of light onto an imaging target in a relative motion relationship to perform imaging. The right side of the figure is a perspective view of the imaging target being scanned by the line of light. As shown on the left side of the figure, the measurement device 200 includes a projection assembly 210, an image sensor 220, and a processor 230.

[0032] The projection assembly 210 is used to project a line of light onto an imaging target that is in relative motion. In the example shown in the figure, the imaging target is moving uniformly in a first direction (x direction in the figure) at a first speed. The length direction of the line of light is a second direction (y direction in the figure) perpendicular to the first direction. The z direction is defined as a direction perpendicular to both the x and y directions, i.e., the vertical direction in the left-hand figure. In the example shown in the figure, the emission direction of the line of light projected by the projection assembly 210 is the z direction, i.e., the line of light is projected vertically downward. In other embodiments, the second direction, which is the length direction of the line of light, may not be perpendicular to the direction of relative motion (as long as it is at a predetermined angle, i.e., it does not have to be parallel). Similarly, the emission direction of the line of light may not be the z direction (as long as it is at a predetermined angle, i.e., it does not have to be parallel). Hereinafter, further explanation will be provided with reference to the examples of FIGS. 5A and 5B.

[0033] In the example of FIG. 2 , the relative motion between the line light and the imaging target is the relative motion between the depth data measurement device itself and the imaging target. The projection angle of the line light from the projection assembly 210 does not move relative to the other assemblies and housing of the depth data measurement device; i.e., the projection assembly 210 does not include a reflecting mechanism that rotates to cause the line light to scan within a predetermined angle. The line light projected by the projection assembly 210 can image the entire imaging target (e.g., items on a conveyor belt) because there is relative motion between the depth data measurement device and the imaging target. (In contrast, in the example of FIG. 7 , the line light projected by the projection assembly 710 itself moves relative to the other assemblies and housing of the depth data measurement device; i.e., the projection assembly 710 includes a rotation mechanism that rotates to cause the line light to scan within a predetermined angle, as shown, for example, in FIGS. 11A-B and 12 .)

[0034] In different application scenarios, the relative motion between the depth data measuring device and the imaging object may be such that the depth data measuring device is stationary and the imaging object is moving, or the imaging object is stationary and the depth data measuring device is moving, or both are moving but not in the same direction or at the same speed, i.e., relative motion.

[0035] As can be seen from this, relative motion is the relative motion of the imaging object relative to the depth data measuring device, and when the depth data measuring device is stationary, the motion information of the imaging object is obtained as the relative motion information, when the imaging object is stationary, the motion information of the depth data measuring device is obtained as the relative motion information, or when both the imaging object and the depth data measuring device are in a moving state, the relative motion information of the imaging object and the depth data measuring device is obtained as the relative motion information.

[0036] Below, the form of motion of the imaging object will be explained with reference to the means for detecting volume information of the conveyor belt article, and the form of motion of the depth data measuring device will be explained with reference to the means for detecting the surface state of the object.

[0037] In the example of FIG. 2, the imaging target is an object being transported on a conveyor belt. The object has an irregular rectangular shape (it can be considered as two rectangles of the same length but different heights and widths put together). The relative motion between the measurement device 200 and the imaging target is caused by the horizontal movement of the conveyor belt. That is, the measurement device 200 can be installed under a cantilever above the conveyor belt and projects a line of light downward. The line of light projected onto the surface of the imaging target is shown as a dot in the left-hand view of FIG. 2 (because the left-hand view of FIG. 2 is a plan view seen from the y direction) and as a line in the oblique view on the right-hand view. Although the line of light is irradiated onto the top surface of the imaging target, due to the difference in height between the top surface of the imaging target and the surface of the conveyor belt and the presence of a certain baseline distance between the projection assembly 210 and the image sensor 220, even if the projection assembly 210 projects the line of light vertically downward, the image displayed on the image sensor 220 remains a broken line due to the projection angle and the actual parallax. There is a difference in height between the line reflected by the top surface of the article and the line reflected by the conveyor belts on both sides. In this case, the height (H) and width (W) of the current position of the imaged object can be determined based on the positions of the lit pixels on the imaged image. Specifically, the height of the current position of the imaged object can be derived based on the difference in height between the center line and the line on both sides, and the pixel positions at which the center line and the line on both sides are located. The width corresponding to the current position of the imaged object can be derived based on the width of the center line and the pixel position of the center line.

[0038] Because only a line of light is projected, few pixels in the resulting image actually contain depth information. For example, the image sensor 220 is shown as an 8x9 = 72-pixel sensor. When imaging the reflected line of light, only nine pixels (i.e., one row of pixels) are illuminated. In a practical application scenario, the image sensor 220 typically has a higher-performance pixel configuration, e.g., 600x800 pixels, and the reflected line of light may occupy more rows (e.g., 3x800 = 2400 pixels illuminated) or columns (depending on the imaging direction), but most pixels in the image still do not contain any depth information. However, when using a typical image sensor, a processor must perform full processing on, e.g., a 600x800-pixel image to extract the desired depth information from it. Because only 3x800 pixels of the 600x800 pixels contain useful information, conventional depth information processing methods significantly waste the processor's computing power.

[0039] For this reason, the present disclosure uses a novel image sensor. Specifically, the image sensor 220 is used to image the target based on the reflected light of the line light, similar to a conventional image sensor. However, among the multiple pixels included in the image sensor, each pixel can transmit a valid signal with a timestamp when the change in the amount of light received within a unit time exceeds a threshold. In the example shown in FIG. 2, the current time t i In this example, only @4x1, @4x2, @2x3, @2x4, @2x5, @2x6, @2x7, @4x8, and @4x9 are projected, and the change in the amount of light received within a unit time exceeds the threshold, so these nine pixels can send valid signals with the current timestamp attached.

[0040] In this case, the unit time may be a very short time, for example, a time period of 1 μs, and each pixel in the same image sensor can calculate the unit time according to the same start time and end time. The threshold can be set according to the application scenario, but it is required to be within the upper and lower limits that the image sensor can realize. If the image sensor 220 actually has, for example, an 800x600 pixel array, each of the 480,000 pixels can independently generate and transmit a valid signal with a time stamp.

[0041] According to the imaging principle of the image sensor 220, in one embodiment, the projected line of light may be a constantly lit laser, and the pixels that generate valid signals at the rising (from absent to present) and falling (from present to absent) edges of the line of light may be laser pulses. The repetition frequency and pulse duration of the laser pulses may be reasonably selected according to the relative speed, the threshold, and the relative distance of the imaging object. In one embodiment, the projected line of light may be high-frequency pulsed light that repetitions 20,000 times per second, thereby supporting high-speed surface defect detection.

[0042] After acquiring the valid signal, the processor 230 can calculate surface depth information of the imaging object based on the position of the pixel corresponding to the valid signal, the timestamp of the valid signal, and the relative motion information. The surface depth information of the imaging object can be used to acquire volume information of the imaging object, such as length, width, and height information.

[0043] For example, in the example of Figure 2, if the conveyor belt moves at a speed of 0.5 meters per second and the depth measurement device continues to acquire valid signals within the timestamps from 0 to 1 second, it can be determined that the length L of the imaging object is 0.5 meters. Furthermore, as described above, the height (H) and width (W) of the current position of the imaging object can be determined based on the positions of the pixels reporting valid signals corresponding to each timestamp. In the example of Figure 2, the heights (H) of the front and rear of the imaging object are different, which is reflected as different height differences between the middle pixel and the pixels on either side that report valid information.

[0044] When performing depth measurements on an imaging object, the depth data measurement device of the present disclosure generally needs to know relative motion information values, such as velocity values or acceleration values, for example, to calculate the volume of the imaging object or to perform corresponding depth annotations for specific imaging positions on the imaging object. In this case, the depth data measurement device must further include an information acquisition device for acquiring the relative motion velocity values. In one embodiment, the motion information acquisition device may be a communication assembly used to acquire the relative motion velocity values from an external source, such as acquiring the current motor rotation speed of a conveyor belt. In another embodiment, when used in a scene where the imaging object is stationary and the depth data measurement device is in a moving state, the motion information acquisition device itself may be an accelerometer used to measure the acceleration of the measurement device, for example, at its location, thereby acquiring the motion velocity of the depth data measurement device itself.

[0045] The example in FIG. 2 shows a monocular depth data measurement device, i.e., the measurement device includes only one image sensor. In a monocular setup, an additional calibration operation of a reference image is required. In another embodiment, the image sensor includes a first image sensor and a second image sensor installed on either side of the projection assembly, and the processor calculates depth information of the imaging object using pixel positions corresponding to valid signals from the first image sensor and the second image sensor, timestamps of the valid signals, and the first relative motion speed. In a binocular configuration, depth calculations are performed using the parallax of the signals included in the first image sensor and the second image sensor, eliminating the need for a calibration operation for a reference image, thereby providing a wider range of applications.

[0046] The depth data measurement device of the present disclosure can be used in various scenes where there is a relative displacement between the measurement device and the imaging target. Figure 3 is a schematic flowchart of a method for measuring volume information of an item on a conveyor belt according to one embodiment of the present disclosure. The method can be performed by the depth data measurement device of the present disclosure as described above.

[0047] In step S310, a line of light is projected toward an object on a conveyor belt. In step S320, the object is imaged based on the reflected light of the line of light, and a valid signal with a timestamp is obtained. In step S330, depth information of the object corresponding to the timestamp is calculated based on the pixel position of the valid signal. In step S340, motion information of the conveyor belt is obtained as motion information of the object. In step S350, volume information of the object is calculated based on the depth information of the object and the motion information of the object at multiple timestamps.

[0048] It is noted that a certain amount of time is required for an object to pass through the projection area of the line light. For example, if a conveyor belt is moving at 0.5 meters per second and a 25 cm long package is placed upright, it will take 0.5 seconds for the line light to complete scanning. In other words, the difference between the time when the leading edge of the package passes through the line light and the time when the trailing edge of the package passes through the line light is 0.5 seconds. In other words, the depth data measurement device of the present disclosure requires at least the time it takes for the object to pass through the line light in order to measure the volume of the object.

[0049] For ease of explanation, FIG. 4 illustrates an embodiment of measuring information about an item on a conveyor belt using a depth data measurement device according to the present disclosure. As shown in FIG. 4, a depth data measurement device 410 is attached to a cantilever 440 and is used to image an item 430 on a conveyor belt 420. In the illustrated example, the depth data measurement device 410 projects a line of light in the z direction (i.e., vertical direction). Measurements of three items (e.g., three parcel boxes) on the left have been completed, and measurement of the following fourth item is about to begin. Because no items are passing through the current projection position, the projected line of light appears as a single straight line. Correspondingly, pixels reporting valid signals in the image sensor of the measurement device 410 should be located in a specific row (or a specific column, depending on the orientation). Only pixels in a specific row reporting valid signals can identify that an item on the conveyor belt has not been measured. Only when pixels reporting valid signals in the image sensor of the measurement device 410 are located in a different row, can the length, width, and height of the corresponding item be calculated.

[0050] In a preferred example shown in Figure 4, the conveyor belt movement direction is a first direction, and the length direction of the line of light is a second direction perpendicular to the first direction. The projection direction of the line of light is perpendicular to the conveyor belt surface or a tangent to the conveyor belt surface (when the conveyor belt has a constant curvature). In other examples, the length direction or projection direction of the line of light may be in another direction.

[0051] 5A-B show an example of measuring information about an item on a conveyor belt using a depth data measurement device. In FIG. 5A, the length direction of the line light is still a second direction perpendicular to the first direction in which the conveyor belt moves. However, the projection direction of the line light is not perpendicular to the conveyor belt surface, but rather has a certain included angle α with the z direction. In this case, a corresponding transformation can be performed based on the included angle to calculate the length, width, and height of the item. Similarly, in FIG. 5B, the projection direction of the line light is not perpendicular to the conveyor belt surface (i.e., has an included angle α with the z direction), and the length direction of the line light is also not perpendicular to the first direction in which the conveyor belt moves, but has an included angle β with the y direction. In this case, when calculating the length, width, and height of the item, a transformation based on both the values of α and β is required.

[0052] Up to this point, we have shown examples in which the measurement device is fixed and the imaging target is moving. The measurement device of the present disclosure can also be applied to scenes in which the imaging target is fixed and the measurement device itself is moving.

[0053] 6 is a schematic flowchart of a method for detecting the surface condition of an object according to one embodiment of the present disclosure, which can be performed by the depth data measurement device of the present disclosure as described above.

[0054] In step S610, a line of light is projected toward the object surface. In step S620, the object surface is imaged based on the reflected light of the line of light, and a valid signal with a timestamp is obtained. In step S630, depth information of the object surface corresponding to the timestamp is calculated based on the pixel position of the valid signal. In step S640, motion information (e.g., acceleration information) of the exercise equipment to which the depth data measurement device is attached is obtained. Here, the motion information (e.g., acceleration information) can be used to derive the road surface position corresponding to each timestamp. In step S650, the state of the object surface position at each timestamp is determined based on the depth information of the object surface at multiple timestamps and the corresponding motion information.

[0055] In one embodiment, the object surface may be a road surface, and the depth data measuring device may be attached to the bottom of the test vehicle to measure the road surface conditions during the vehicle's travel.

[0056] In one embodiment, the road surface may be the surface of a highway. In this case, the measurement device may measure the flatness of the measured road surface. For example, normally, pixel locations reporting valid signals should be located in the same row or in several adjacent rows. If several pixel locations reporting valid signals are in higher or lower rows, this indicates the presence of a hole or protrusion on the road surface, and repairs are required. In this case, a timestamp corresponding to the valid signal of the hole or protrusion is recorded, and the road surface position corresponding to the timestamp can be calculated based on speed information (e.g., acceleration information measured by an accelerometer), and then repairs can be performed.

[0057] In one embodiment, the road surface is a railway surface having a predetermined depth distribution pattern. The depth data measurement device of the present disclosure can be attached to the bottom of a train or a dedicated track test vehicle to detect road surface conditions. In this case, determining the road surface condition at a road surface position corresponding to each timestamp based on the road surface depth information at multiple timestamps includes comparing the road surface depth information at the multiple timestamps with corresponding road surface position depth information in the predetermined depth distribution pattern to determine the road surface condition, and / or synthesizing an actual road surface depth distribution based on the road surface depth information at the multiple timestamps and the road surface positions corresponding to each timestamp, and determining the road surface condition based on the actual road surface depth distribution. Because railway surfaces have undulations such as sleepers, when detecting the condition of the railway surface, it is necessary to determine whether the surface distribution synthesized based on the depth information matches the surface distribution of a railway.

[0058] The depth data measurement device of the present disclosure can be used to detect surface defects of other objects, particularly large objects (e.g., large ships). The surface of the object to be measured may have a predetermined contour. In this case, the condition of the object surface can be determined by comparing depth information of the object surface at multiple time stamps with the predetermined contour of the corresponding object surface location. In another example, if the surface of the object to be measured should have a smooth configuration, the presence of holes or protrusions can be determined at pixel locations to detect defects.

[0059] 2 to 6, a measurement method and application scenario using a depth data measurement device have been described in which the relative motion between the line light and the imaging target is caused by the relative motion between the depth data measurement device and the imaging target. In another embodiment, the relative motion between the line light and the imaging target may be realized by a projection assembly projecting a line light that moves within a predetermined angle. In this case, the line light is scanned and projected onto the imaging target by rotation (e.g., reciprocating motion within a predetermined angle) of a rotation mechanism included in the projection assembly, and the line light projected by the projection assembly itself moves relative to the other assemblies and housing of the depth data measurement device, and there is no relative motion between the other assemblies and housing of the depth data measurement device and the imaging target.

[0060] 7 is a schematic diagram of a depth data measurement device according to one embodiment of the present disclosure. As shown in the figure, a measurement head 700 includes a projection assembly 710, an image sensor 720, and a processor 730.

[0061] The projection assembly 710 is used to project a line of light moving along a first direction (shown as the x-direction in the figure) onto the imaging area, and the length direction of the line of light is preferably a second direction (shown as the y-direction in the figure) perpendicular to the first direction. As shown in the figure, the line of light generated by the projection assembly 710 is emitted in the z-direction (and within its scanning range, e.g., within a range of ±10° in the z-direction) and can be projected into the imaging area of the image sensor. In one embodiment, before deriving depth data of an imaging object in the imaging area, an alignment operation can be performed first, for example, to align the scanning projection range of the projection assembly 710 with the imaging area of the image sensor 720. The alignment operation can be performed using a calibration plane. For example, if no imaging object is placed, a relatively flat plane (e.g., a wall surface perpendicular to the z-direction) in the imaging area can be used as the calibration plane, and a stripe pattern can be scanned and imaged using the image sensor. The scan angle of the projection assembly 710 and / or the imaging field of view of the image sensor 720 can be adjusted so that the image sensor can image one complete projected image, and the projected image covers substantially the entire imaging range of the image sensor.

[0062] Similarly, in the example of FIG. 7, the image sensor is not a traditional global shutter or rolling shutter sensor, but rather includes pixels that are capable of generating and transmitting a time-stamped valid signal when the amount of transmitted light received by that pixel within a unit of time exceeds a threshold.

[0063] In this case, the unit time can be a very short time, for example, a time period of 1 μs, and each pixel in the same image sensor can calculate the unit time according to preferably the same start time and end time. The threshold can be set according to the application scenario, but it is required to be within the upper and lower limits that the image sensor can realize.

[0064] JPEG2025525974000003.jpg179170

[0065] Therefore, the processor 730 can generate a two-dimensional image of the stripe light pattern based on the stripe light pattern brightness corresponding to the timestamp and the location of the pixel reporting the valid signal, and derive depth information of the imaging object within the imaging region based on the two-dimensional image.

[0066] In the above example, from t0=0 to t 960 In a 0.96 ms time period of =1000 μs, the projection assembly 710 completes one scan of the line light, for example, a left-to-right scan which may correspond to one scan from the start position AA′ of one scan to the end position BB′ of one scan in FIG. 7.

[0067] As described above, if the scanning speed and scanning position of the projection assembly are well matched with the imaging area range corresponding to each pixel column of the image sensor 720, there is no imaging target in the imaging area, and only walls perpendicular to the projection direction exist, and the calibration plane is as shown in FIG. 7, the processor 730 performs the time t1=1 μs to t 960 For 960 time points of t = 960 μs, the effective signal of each pixel row of the image sensor 420 can be received at each time point. For example, the effective signal of the first pixel row is received at time t1, the effective signal of the second pixel row is received at time t2, and by analogy, finally, the effective signal of the second pixel row is received at time t3. 960 The processor receives the valid signal of the last pixel row at time t1. Based on the corresponding timestamps, the processor can reconstruct the pattern shown in pattern 1 of Figure 8. As shown in the above example, pixels with valid signal timestamp t1 are given a brightness corresponding to the stripe light pattern at time t1, and pixels with valid signal timestamp t2 are given a brightness corresponding to the stripe light pattern at time t2. By analogy, a two-dimensional image of the stripe light pattern can finally be obtained.

[0068] However, if an imaging object is included within the imaging area, the projected line of light will change due to reflection from the imaging object, and in this case, pixels reporting valid signals at the same timestamp are not limited to one pixel column.

[0069] As shown in Figure 7, when a sphere is included in the imaging area, the projected line of light is distorted, and this distortion reflects the change in the pixel positions reporting the valid signal, which allows the depth information of the imaged object to be determined.

[0070] For clarity, Figure 9 shows an example of line light projection and imaging when an imaging target is absent and present in the imaging region. The top of Figure 9 shows the pattern that projection assembly 710 intends to project (or, in a different embodiment, the pattern that processor 730 can reconstruct), and the bottom of Figure 9 shows pixel locations reporting valid signals at specific times resulting from scanning the line light during the projection process.

[0071] Specifically, the left side of Figure 9 shows a case where there is no imaging object in the imaging region (e.g., there is only one calibration wall perpendicular to the z direction), while the right side shows a case where the imaging region contains one imaging object (e.g., the sphere shown in Figure 7).

[0072] In one embodiment of the present disclosure, the projection assembly 710 can continuously change the intensity of the line light during one scan, for example, from the scan start position AA′ to the scan end position BB′ shown in FIG. 7, thereby realizing the projection of a stripe structured light pattern as shown in the upper left of FIG. 9 during one scan period T. As described above, when the projection assembly 710 is operated from t0=0 to t 960In the example where one scan of the line light is completed within a time of 0.96 ms, t = 960 μs (for convenience of explanation, it can be assumed that after the projection assembly completes one scan, it takes 40 μs to prepare for the next scan, and therefore one scan period can be set as T = 960 μs + 40 μs = 1000 μs = 1 ms), 480 At this time, the projection assembly 710 scans to the center of the imaging area, and the brightness of the line light is the brightest, as shown in the lower left of Figure 9. At this time, if there is no imaging object in the imaging area, the pixels in the 480th column corresponding to the image sensor 720 will experience a change in light intensity within a unit time, and 480 If the imaging region contains a spherical imaging target as shown on the right side of Figure 9, t 480 At this time, not all pixels in the 480th row of the image sensor 720 experience a change in light intensity within the unit time, but only the upper and lower pixels of the 480th row that are not blocked by the imaging object experience a change in light intensity within the unit time. 480 and reports on@t in the center of the pixel array. 480 is located before the pixel in the 480th column. Therefore, the depth data measurement device of the present disclosure aligns the projection brightness with the pixel reporting the valid signal based on the timestamp, thereby causing the processor 420 to reconstruct a two-dimensional stripe image, such as that shown in the top right of FIG. 9, that includes depth information of the imaging object when the imaging region includes a spherical imaging object.

[0073] As described above, the processor 730 can generate a two-dimensional image of the stripe light pattern based on the stripe light pattern brightness corresponding to a timestamp and the pixel locations reporting valid signals, and derive depth information of the imaged object within the imaging region based on the two-dimensional image. In some embodiments, the stripe light pattern brightness corresponding to a timestamp can be the brightness of the stripe image actually projected by the projection assembly 710 at the corresponding timestamp. In other embodiments, because each pixel of the image sensor 720 reports a valid signal only when the change in light flux within a unit time is greater than a threshold (in other words, the valid signal itself cannot characterize the light intensity of the projected line light itself), the stripe light pattern brightness corresponding to a timestamp can simply be the brightness corresponding to a predetermined stripe light pattern, rather than the brightness of the stripe image actually projected by the projection assembly 710 at the corresponding timestamp (in this case, the projection assembly 710 only needs to ensure that the corresponding pixel reports a valid signal by scanning the line light).

[0074] First, we will describe an easily understood embodiment in which the stripe light pattern brightness corresponding to a timestamp is the stripe image brightness actually projected by the projection assembly 710 at the corresponding timestamp.

[0075] As described above, encoding of stripe structured light typically requires projecting different stripe light patterns multiple times and encoding pixels of the multiple patterns to derive depth information of the imaging object. For example, in the example of time code stripe structured light shown in Figures 1A-1B, five different stripe light patterns must be projected consecutively to generate one depth image. In the example of four-step phase-shifted sinusoidal stripe structured light shown in Figure 8, four different stripe light patterns must be projected consecutively to generate one depth image.

[0076] Therefore, in one embodiment of the present disclosure, the projection assembly 710 is used to complete one pattern scan within one scan period T, and within the scan period T, the projection assembly adjusts the brightness of the line light in moving projection based on one stripe light pattern to be projected, and the projection assembly adjusts the brightness of the line light based on different stripe light patterns within N consecutive scan periods T, and the processor generates N two-dimensional images corresponding to the N stripe light patterns based on the N scan periods T, and synthesizes one depth image of the imaging object in the imaging area based on the N two-dimensional images.

[0077] Similarly, taking the pattern shown in FIG. 8 as an example, the projection assembly 710 can project Pattern 1 during a first scanning process from the scan start position AA′ to the scan end position BB′ shown in FIG. 7 , project Pattern 2 during a second scanning process from AA′ to BB′, project Pattern 3 during a third scanning process from AA′ to BB′, and project Pattern 4 during a fourth scanning process from AA′ to BB′. Each projection corresponds to a projection period T. If T=1 ms, it takes 4 ms to project four patterns. For each projection, the line light can change its brightness during the projection process based on the corresponding stripe light pattern, combine an appropriate projection speed, and assign brightness values to corresponding pixel positions based on the timestamp of the valid signal. After four projections, four two-dimensional images are synthesized by the processor, and a depth image of the imaging object within the imaging region is synthesized based on the four two-dimensional images.

[0078] In another embodiment of the present disclosure, multiple patterns may be projected in one scan, in which the projection assembly completes one pattern scan within a scan period T, during which the projection assembly alternately adjusts the brightness of the line light in the moving projection based on the N stripe light patterns to be projected, and the processor generates N two-dimensional images corresponding to the N stripe light patterns based on the one scan period T, and synthesizes one depth image of the imaging object in the imaging area based on the N two-dimensional images.

[0079] JPEG2025525974000004.jpg51170

[0080] where Q is the brightness of the brightest point of the pattern and θ is the angle of the four-step phase shift corresponding to time t.

[0081] JPEG2025525974000005.jpg42170

[0082] JPEG2025525974000006.jpg150170

[0083] In an example where the projection assembly 710 is used to perform multiple pattern scans within one scan period T, projections with varying intensity at the same position are involved. As shown in FIG. 10, when t0=0, P1=Q / 2; P2=0; P3=Q / 2; P4=Q. Therefore, if the unit time for reporting a valid signal of the image sensor is 1 μs as described above, four projections with varying intensity are required at the starting position to complete the conversion projections for the four patterns. Therefore, t 0_1 When t = 0, the light intensity of the line light is Q / 2, and 0_2 When t = 1 μs, the light intensity of the line light is 0. 0_3 When t = 2 μs, the light intensity of the line light is Q / 2, and 0_4When t1 = 3 μs, the light intensity of the line light can be Q. When t1 = 4 μs (corresponding to θ = 2π / 80), P1 = 0.54 * Q; P2 = 0.02 * Q; P3 = 0.46 * Q; P4 = 0.998 * Q. Therefore, t 1_1 When t = 4 μs, the light intensity of the line light is 0.54*Q, and 1_2 When t = 5 μs, the light intensity of the line light is 0.02*Q, and 1_3 When t = 6 μs, the light intensity of the line light is 0.46*Q, and 1_4 When Q = 7 μs, the light intensity of the line light can be 0.998*Q. Here, a sub-period in which one pattern intensity change is performed every 4 μs may be referred to as one projection sub-cycle. Within one projection sub-cycle, the projection assembly switches between N different intensities at the same position corresponding to N patterns. Therefore, from t0 to t 960 Each time point further includes four sub-time points, and it takes 3840 μs (=960 μs×4) to complete 960 projections of each of the four patterns. In this embodiment, the projection assembly 710 alternately projects line lights at intensities corresponding to N positions of the stripe light pattern, and the processor corresponds the pixel positions of the effective signals to the projection intensities of the line lights based on the timestamps, and classifies them into N categories corresponding to each stripe light pattern (for example, a timestamp divisible by 4 is regarded as the timestamp corresponding to the first pattern, a timestamp that is divided by 4 with a remainder of 1 is regarded as the timestamp corresponding to the second pattern, a timestamp that is divided by 4 with a remainder of 2 is regarded as the timestamp corresponding to the third pattern, and a timestamp that is divided by 4 with a remainder of 3 is regarded as the timestamp corresponding to the fourth pattern), thereby generating N two-dimensional images corresponding to the N stripe light patterns, and synthesizing one depth image of the imaging object in the imaging area based on the N two-dimensional images.

[0084] In one embodiment, within each projection sub-cycle, the projection position of the line of light may remain unchanged. For example, the projection assembly may remain stationary for 4 μs at the corresponding position of each pixel column, thereby completing brightness switching corresponding to four patterns, then move to the next position and remain stationary for 4 μs again, and so on. In another embodiment, within one projection sub-cycle, the distance traveled by the line of light projected by the projection assembly across the imaging area is equal to or less than the corresponding width of a single pixel column. In other embodiments, different pixel columns may be selected to correspond to different patterns (in which case precise calibration of the scanning position and the imaging area is required).

[0085] In the above-mentioned means for scanning multiple patterns at one time, the N stripe light patterns are patterns in which there are no pixels of the same brightness in the same position between adjacent patterns, and are, for example, four sine wave four-step phase shift patterns shown in Figures 8 and 10.

[0086] As described above, because the valid signal reported by each pixel of the image sensor 720 cannot characterize the light intensity of the currently projected line light, the brightness of the stripe light pattern corresponding to a timestamp is not the brightness of the stripe image actually projected by the projection assembly 710 at the corresponding timestamp. In this case, the projection assembly 710 can complete one scan within one scan period T, and the processor 730 can assign brightness values corresponding to the pre-specified stripe light patterns to pixel positions corresponding to the timestamps. Specifically, the processor assigns N brightness values corresponding to the N pre-specified stripe light patterns to pixel positions corresponding to the timestamps, thereby generating N two-dimensional images corresponding to the N stripe light patterns, and synthesizing a depth image of the imaging object in the imaging region based on the N two-dimensional images.

[0087] Here, we again take an example of deriving depth data of an imaged object based on the four sinusoidal four-step phase-shift patterns shown in FIGS. 8 and 10 . Assume that the projection assembly 710 completes one scan from position AA′ to position BB′ within 960 μs. At this time, the projection assembly 710 can perform line light scanning at any brightness at which the image sensor 720 can generate a valid signal from the scanned pixels. For example, the projection assembly 710 can project a line light at a constant brightness equal to or greater than a threshold brightness, and report a valid signal with a timestamp when each pixel of the image sensor 720 is scanned by the line light. The processor 730 can then construct a corresponding two-dimensional image based on a stripe encoding algorithm (e.g., the time code shown in FIG. 1B , the sinusoidal four-step phase-shift code shown in FIG. 8 , or a square wave four-step phase-shift code or Gray code, not shown). When a 960×720 pixel image sensor 720 is used, the processor 730 can generate a corresponding two-dimensional image based on the timestamps t0 to t 960The valid signals from each pixel up to a certain point can be obtained. Therefore, the processor 730 divides a preset pattern (for example, pattern 1 among the 4-step phase shift stripe light patterns shown in FIG. 8) into 960 equal parts, obtains 960 luminance distribution values in its first direction, and assigns those luminance values to the pixel positions corresponding to the timestamps respectively to construct the first two-dimensional image. Next, the processor 730 divides pattern 2 among the 4-step phase shift stripe light patterns into 960 equal parts, obtains 960 luminance distribution values in its first direction, and assigns those luminance values to the pixel positions corresponding to the timestamps respectively to construct the second two-dimensional image. The processor 730 divides pattern 3 among the 4-step phase shift stripe light patterns into 960 equal parts, obtains 960 luminance distribution values in its first direction, and assigns those luminance values to the pixel positions corresponding to the timestamps respectively to construct the third two-dimensional image. The processor 730 divides pattern 4 among the 4-step phase shift stripe light patterns into 960 equal parts, obtains 960 luminance distribution values in its first direction, and assigns those luminance values to the pixel positions corresponding to the timestamps respectively to construct the fourth two-dimensional image. Thus, based on the valid signals with the timestamps reported by each pixel obtained in one scan of the projection assembly, the processor 730 can construct any number of two-dimensional images (usually a stripe light image set following a specific encoding algorithm, for example, the four sine wave 4-step phase shift stripe light images shown in FIG. 8) based on the luminance at the corresponding positions of a predetermined image, and thereby calculate the depth data information of the imaging target, greatly improving the calculation efficiency.

[0088] In one embodiment, as shown in FIG. 7, the projection assembly shown in FIG. 2 preferably projects line light pulses instead of a specific stripe light pattern, for example, pulses that blink 20,000 times per second.

[0089] Furthermore, as will be understood, for ease of understanding, the above example assumes an ideal situation in which an ideal line light (i.e., an ideal "line segment" with no width) scans a width corresponding to one column of pixels during a unit. However, in actual operation, the line light extending along the second direction often has a specific width (in the first direction) and may have an uneven distribution of brightness in the width direction. Each pixel of the image sensor reports a valid signal when the change in light amount within a unit time exceeds a threshold (either an increase in light amount or a decrease in light amount exceeds the threshold). Therefore, the scanning of the trailing edge of the line light or a change in light intensity within the line light can both cause a pixel to report a valid signal. Therefore, when a pixel row of the image sensor forms an included angle with respect to the second direction, preferably perpendicular to the second direction, i.e., parallel to the first direction, and there are multiple pixels in the same pixel row that send valid signals with the same timestamp, the processor 730 filters the pixels to be used in generating the two-dimensional image based on the light intensity distribution of the line light in the first direction. For example, processor 730 may simply select the valid signal that reported the passage of the leading edge of the line light and ignore subsequent valid signals (e.g., if the line light causes multiple adjacent pixels on the same pixel row to report valid signals with the same timestamp, only the valid signal reported by the foremost pixel may be retained).

[0090] In addition, if there are multiple pixels in the same pixel row that send valid signals with the same timestamp, the processor may delete pixels in the pixel row used to generate the two-dimensional image that are far from the current scan range. For example, immediately after a line of light enters one side of the imaging area of the image sensor, valid signals reported by pixels on the other side can be ignored, thereby avoiding the influence of noise signals.

[0091] Also, if a single pixel transmits multiple valid signals with consecutive timestamps, the processor selects one of the consecutive timestamps as the timestamp for the corresponding valid signal of the pixel. For example, the processor may select the first timestamp or the last timestamp based on a specific algorithm. This is because the line light in actual operation is not an ideal line, and for example, the width or brightness varies.

[0092] In the example shown in FIG. 7, the depth data measurement head of the present disclosure may include only a single image sensor 720, while in other embodiments, the depth data measurement head of the present disclosure may include two image sensors, each including a plurality of pixels, and each pixel transmitting a valid signal with a timestamp when the change in the amount of light received within a unit time exceeds a threshold, thereby realizing a dual-lens installation.

[0093] Furthermore, the projection assembly of the present disclosure can achieve line-light scanning projection by rotating the rotation mechanism. Figures 11A-B are enlarged views of the projection assembly shown in Figure 7. Specifically, in the projection assembly 710, a laser emitted by a laser light-emitting element (e.g., laser light-emitting element 711) is scanned and projected onto a calibration plane (the gray area in Figure 7) via a rotation mechanism (e.g., projection mechanism 712) to actively project structured light onto an imaging target (e.g., a sphere in Figure 7) within the imaging area. In practical applications, the laser light-emitting element 711 is used to generate a line infrared laser. In various embodiments, the laser light-emitting element 711 can perform high-speed switching to scan and project alternating bright and dark structured light corresponding to a stripe code. The high-speed switching may include high-speed switching of the laser light-emitting element and high-speed code switching. Alternatively, the laser light-emitting element 711 may perform scanning while maintaining a constant brightness without switching brightness (in this case, the processor 730 assigns a brightness value based on a predetermined position as described above).

[0094] In one embodiment, the laser light emitting element 711 may be a line laser light emitting element that generates a line of light extending in the y direction (a direction perpendicular to the paper surface in FIGS. 11A-B). The line of light is then reflected by a rotation mechanism 712 that can swing along an axis in the x direction and projected onto an imaging plane. The swing diagram of the rotation mechanism 712 is shown in FIG. 11B. This allows for a reciprocating line of light scanning within the range AB (the range from AA' to BB').

[0095] In one embodiment, the rotation mechanism 712 may be a micromirror device (also referred to as a digital micromirror device, DMD), and may be realized as a MEMS (microelectromechanical system). Fig. 12 is a simplified perspective view of the projection assembly used in the present disclosure. As shown in Fig. 12, a spot laser generated by a laser element can be converted into a line light through a lens (corresponding to the line laser light emitting element 411 in Fig. 11), and the line light is further reflected by a MEMS-type micromirror device, and the reflected line light is further projected into external space through an optical window.

[0096] The present disclosure can also be realized as a depth data measurement method. Fig. 13 is a schematic flowchart of a depth data measurement method according to an embodiment of the present disclosure. The method can be performed by the depth data measurement head of the present disclosure.

[0097] In step S1310, a line of light is projected onto an imaging target, and relative motion exists between the line of light and the imaging target. In step S1320, the imaging target is imaged based on the reflected light of the line of light, and each pixel transmits a valid signal with a timestamp when the change in the amount of light received within a unit time exceeds a threshold. In step S1330, surface depth information of the imaging target is calculated based on the pixel position and timestamp reporting the valid signal and the relative motion information.

[0098] The step of projecting a line of light onto the imaging target includes a step of scanningly projecting the line of light onto the imaging target by rotating a rotation mechanism, as shown in Figures 7 to 12, and may also include a step of projecting the line of light in a fixed direction relative to the housing of the depth data measuring device, as shown in Figures 2 to 6, wherein there is relative motion between the depth data measuring device and the imaging target.

[0099] When the line light is scanned and projected onto the imaging target by rotating the rotation mechanism, alignment and calibration can be performed before performing measurement. Therefore, the measurement method disclosed herein may further include a step of aligning the scan projection range of the projection assembly with the imaging area of the image sensor.

[0100] The depth data measurement device and its application method according to the present disclosure have been described in detail above with reference to the drawings. The means uses line light to actively perform optical imaging of an imaging object in the presence of relative motion, and combines an image sensor in which each pixel can independently generate an effective signal when the brightness changes, so that imaging can be performed only on the changed effective signal, and the position of the depth data is determined using timestamps and relative motion information, thereby realizing high-speed acquisition and processing of depth information with a simple projection mechanism and extremely low computing power requirements.

[0101] Although various embodiments of the present disclosure have been described, the above description is illustrative and not exhaustive, and is not limited to the various embodiments disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles, practical applications, or improvements to commercially available technologies of the various embodiments, or to enable those skilled in the art to understand the various embodiments disclosed herein.

Claims

1. 1. A depth data measurement device, comprising: a projection assembly for projecting a line of light onto an imaging target in the presence of relative motion; an image sensor that images the imaging target based on reflected light of the line light, wherein each pixel transmits a valid signal with a timestamp attached when a change in the amount of light received within a unit time exceeds a threshold; a processor for calculating surface depth information of the imaging object based on pixel positions corresponding to the valid signals, timestamps of the valid signals, and relative motion information.

2. the relative motion is motion of the imaging object relative to the depth data measurement device; and When the depth data measurement device is stationary, motion information of the imaging object is acquired as the relative motion information; When the imaging object is stationary, motion information of the depth data measurement device is acquired as the relative motion information; or The depth data measuring device according to claim 1 , wherein when the imaging object and the depth data measuring device are both in motion, relative motion information between the imaging object and the depth data measuring device is obtained as the relative motion information.

3. The depth data measurement device according to claim 2 , further comprising a motion information acquisition device used to acquire the relative motion information.

4. the image sensor includes a first image sensor and a second image sensor located on opposite sides of the projection assembly; 2. The depth data measurement device of claim 1, wherein the processor calculates depth information of the imaging object using the positions of pixels corresponding to the valid signals and timestamps of the valid signals from the first image sensor and the second image sensor, as well as a first relative motion velocity.

5. The depth data measurement device of claim 1 , wherein the projection assembly projects line light pulses at a predetermined frequency.

6. The depth data measurement device of claim 1 , wherein the relative movement is movement of the line light projected by the projection assembly relative to the image sensor and the depth data measurement device, and the projection assembly performs scanning projection with the line light by rotating a rotation mechanism.

7. the projection assembly scan-projects a line of light moving along a first direction onto an imaging area by rotation of a rotation mechanism, wherein a length direction of the line of light is a second direction perpendicular to the first direction, and the scan-projected line of light forms a stripe light pattern by changing brightness; The depth data measurement device of claim 1, wherein the processor is used to generate a two-dimensional image of the striped light pattern based on the brightness of the striped light pattern corresponding to the pixel position and the timestamp corresponding to the valid signal, and to derive surface depth information of the imaging object within the imaging area based on the two-dimensional image.

8. The projection assembly completes one pattern scan projection within a scan period T, and within the scan period T, the projection assembly adjusts the brightness of the line light in the moving projection based on one stripe light pattern to be projected; and the projection assembly adjusts the brightness of the line light based on different stripe light patterns within N consecutive scan periods T; The depth data measurement device of claim 7, wherein the processor generates N two-dimensional images corresponding to N stripe light patterns based on the N scanning periods T, and synthesizes one depth image of the imaging object within the imaging area based on the N two-dimensional images.

9. The projection assembly completes one pattern scan within a scan period T, and within the scan period T, the projection assembly alternately adjusts the brightness of the line light in moving projection based on N stripe light patterns to be projected; The depth data measurement device of claim 7, wherein the processor generates N two-dimensional images corresponding to the N stripe light patterns based on one of the scanning periods T, and synthesizes one depth image of the imaging object within the imaging area based on the N two-dimensional images.

10. 8. The depth data measurement device of claim 7, wherein the projection assembly completes one scan within a scan period T, and the processor assigns a brightness corresponding to a pre-specified stripe light pattern to the pixel location corresponding to the timestamp.

11. Executed by a depth data measurement device according to any one of claims 1 to 5, and projecting a line of light toward an article on a conveyor belt; imaging the article based on the reflected light of the line light and obtaining a valid signal with a time stamp; calculating depth information of the article corresponding to a timestamp based on pixel locations corresponding to the valid signals; acquiring the movement information of the conveyor belt as the movement information of the item; and calculating volume information of the item based on the depth information of the item at multiple timestamps and the motion information of the item.

12. Executed by a depth data measurement device according to any one of claims 1 to 5, and Projecting a line of light toward a surface of an object; imaging a road surface based on reflected light of the line light and acquiring a valid signal with a time stamp attached; calculating depth information of the surface of the object corresponding to a timestamp based on pixel locations corresponding to the valid signals; acquiring the movement information of the depth data measurement device; A method for detecting the surface state of an object, comprising a step of determining the state of the surface position of the object corresponding to each timestamp based on the depth information of the surface of the object at multiple timestamps and the motion information.

13. The step of determining a state of the surface position of the object corresponding to each timestamp based on the depth information of the surface of the object at each timestamp includes: comparing depth information of the surface of the object at multiple timestamps with depth information of surface positions of the object corresponding to a predetermined depth distribution pattern to determine the surface condition of the object; and / or 13. The method of claim 12, comprising a step of synthesizing an actual surface depth distribution of the object based on the depth information of the surface of the object at multiple timestamps and the surface positions of the object corresponding to each timestamp, and determining the state of the road surface based on the road surface depth distribution of the object surface.

14. projecting a line of light onto an imaging target in which relative motion exists; imaging the imaging target based on the reflected light of the line light, and each pixel transmitting a valid signal with a timestamp when a change in the amount of light received within a unit time exceeds a threshold; and calculating surface depth information of the imaging object based on the position of a pixel corresponding to the valid signal, a timestamp of the valid signal, and relative motion information.

15. The step of projecting the line light onto the imaging target further comprises: Scanning and projecting the line light onto the imaging target by rotating a rotation mechanism; 15. The method of claim 14, comprising projecting the line of light in a direction relative to a housing of a depth data measurement device, wherein there is relative motion between the depth data measurement device and the imaging subject.

Images