Distance image device, method for correcting distance image, and image processing program

Abstract

To provide a distance image device capable of accurately performing abnormality detection.SOLUTION: A distance image device 100 includes: a light scanning unit 22 configured to scan light toward an object to be monitored OB; a light receiving unit 23 configured to receive reflected light of the scanning light from the light scanning unit 22; and an abnormality determination unit 52 configured to determine whether an abnormality is present in a judgment target pixel JP of a distance image, the determination being made on the basis of the comparison between the distance value of the judgment target pixel and distance values of adjacent pixels adjacent to the judgment target pixel, the distance image being on the basis of light scanning by the light scanning unit 22 and a light receiving result of the light receiving unit 23.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a distance image device that generates a distance image that measures the distance to a monitoring target based on the reflected component of an electromagnetic wave emitted toward the monitoring target, a distance image correction method, and an image processing program. [Background technology]

[0002] For example, a known distance measurement device detects noise caused by sunlight (see Patent Document 1). Another known device divides distance values ​​into fixed distance units and labels them, regarding labels with a small number of pixels as noise and performing interpolation from surrounding pixels (see Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-181291 [Patent Document 2] Japanese Patent Application Publication No. 11-120359 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, for example, there is a possibility that noise that is at a similar distance to the object to be extracted cannot be accurately removed. Also, in Patent Document 2, noise is determined based on an area extracted by grouping objects that are close to each other, so there is a possibility that normal pixels will also be removed by collectively determining a range of pixels as noise, and conversely, there is a possibility that noise that occurs in a pinpoint cannot be removed.

[0005] The present invention has been made in view of the above-mentioned points, and an object of the present invention is to provide a range imaging device that can accurately perform abnormality detection. [Means for solving the problem]

[0006] To achieve the above objective, the distance imaging device comprises an optical scanning unit that scans light toward a monitored object, a light receiving unit that receives reflected light from the scanned light by the optical scanning unit, and an abnormality determination unit that determines whether or not there is an abnormality in a target pixel in a distance measurement image based on the optical scanning by the optical scanning unit and the light receiving results at the light receiving unit, based on a comparison between the distance value of the target pixel and the distance value of an adjacent pixel adjacent to the target pixel.

[0007] In the above-described distance image device, the presence or absence of an abnormality can be determined based on a comparison between the distance value of the pixel to be determined and the distance value of the adjacent pixel adjacent to the pixel to be determined, thereby enabling accurate abnormality detection on a pixel-by-pixel basis.

[0008] In a specific aspect of the present invention, the optical scanning unit is a Lissajous scanning type, and the anomaly determination unit includes, as neighboring pixels, pixels on a scanning path different from the scanning path of the target pixel. In this case, distance values ​​for the target pixel and the neighboring pixels on the different scanning paths are measured under different conditions, and by comparing these values, accurate anomaly detection becomes possible.

[0009] In another aspect of the present invention, the anomaly determination unit treats eight pixels surrounding the target pixel as neighboring pixels, enabling more accurate anomaly detection on a pixel-by-pixel basis by making a determination based on a comparison with surrounding measurement results.

[0010] In yet another aspect of the present invention, the abnormality determination unit determines that the target pixel has an abnormality when there is a distance difference between the target pixel and a predetermined number or more of the eight pixels, thereby enabling reliable detection of an abnormality.

[0011] In yet another aspect of the present invention, whether or not to determine whether or not an anomaly exists in a current ranging image is determined based on whether or not a time difference inversion anomaly exists in a past or current ranging image between multiple ranging images acquired consecutively during a predetermined period. In this case, detection based on the passage of time becomes possible.

[0012] In yet another aspect of the present invention, when a time difference inversion anomaly is detected, the anomaly determination unit treats pixels in the current ranging image within a range expanded by a predetermined number of pixels from the position where the time difference inversion anomaly occurred as pixels to be determined. In this case, an anomaly can be detected within an appropriate range.

[0013] In yet another aspect of the present invention, a correction unit is provided that corrects the distance value in pixel units in accordance with the determination result of the abnormality determination unit. In this case, the location where an abnormality is found can be corrected.

[0014] A distance image correction method for achieving the above-mentioned object includes a distance value calculation process for calculating a distance value on a pixel-by-pixel basis based on the optical scanning performed by the optical scanning unit and the light reception results performed by the light receiving unit, a distance value comparison process for comparing the distance values ​​of the pixel to be judged and the adjacent pixels adjacent to the pixel to be judged in the distance measurement image generated by the distance value calculation process, and a distance measurement value correction process for determining whether or not there is an abnormality in the distance measurement image based on the comparison results in the distance value comparison process and correcting the distance value on a pixel-by-pixel basis in accordance with the judgment result.

[0015] In the above-described distance image correction method, the presence or absence of an abnormality is determined based on a comparison between the distance value of the pixel to be judged and the distance value of an adjacent pixel adjacent to the pixel to be judged, thereby detecting abnormalities on a pixel-by-pixel basis and enabling accurate correction of the distance image.

[0016] The image processing program for achieving the above-mentioned objective executes a distance value calculation process that calculates a distance value on a pixel-by-pixel basis based on the optical scanning performed by the optical scanning unit and the light reception results performed by the light receiving unit, a distance value comparison process that compares the distance values ​​of the pixel to be judged and the adjacent pixels adjacent to the pixel to be judged in the ranging image generated by the distance value calculation process, and a ranging value correction process that judges whether or not there is an abnormality in the ranging image based on the comparison results from the distance value comparison process and corrects the distance value on a pixel-by-pixel basis according to the judgment result.

[0017] In the image processing program, the presence or absence of an abnormality is determined based on a comparison between the distance value of the pixel to be judged and the distance value of an adjacent pixel adjacent to the pixel to be judged, thereby detecting abnormalities on a pixel-by-pixel basis and enabling accurate correction of the distance image. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram showing an example of the configuration of a distance image device according to an embodiment; [Figure 2] FIG. 1 is a conceptual diagram showing a scanning process using a Lissajous scan. [Figure 3] FIG. 1A is a conceptual diagram for explaining how distance measurement is performed under normal circumstances, and FIGS. 1B and 1C are conceptual diagrams for explaining how distance measurement is performed under abnormal circumstances. [Figure 4] (A) is a conceptual image diagram showing an example of a target, (B) is a conceptual image diagram showing a distance image of the target under normal conditions, and (C) is a conceptual image diagram showing a distance image of the target under abnormal conditions. [Figure 5] 10A to 10D are conceptual diagrams for explaining the characteristics of scanning by Lissajous scanning. [Figure 6] FIG. 10 is a conceptual diagram for explaining an example of a part of a procedure for specifying a determination target pixel. [Figure 7] 10A to 10C are conceptual diagrams for explaining an example of a process for identifying invalid pixels (abnormal ranging pixels) from one frame of a ranging image. [Figure 8] 10A to 10C are conceptual diagrams illustrating an example of a method for selecting a determination target pixel and performing an abnormality detection (abnormality determination) on the selected determination target pixel. [Figure 9] 10 is a flowchart illustrating a series of processes for correcting a distance image. [Figure 10] 10 is a flowchart illustrating a series of processes for correcting a distance image. [Figure 11] 10 is a flowchart illustrating a series of processes for correcting a distance image. [Figure 12] 10 is a flowchart illustrating a series of processes for correcting a distance image. [Figure 13] FIG. 1 is a conceptual diagram showing an overview of abnormality determination of a distance image by a distance imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of a distance imaging device according to an embodiment will be described below with reference to FIG. 1 and other figures. As illustrated in the block diagram of FIG. 1, the distance imaging device 100 of this embodiment includes a distance measurement unit 10 and an image processing unit 50. It measures distance by projecting laser light, which is an electromagnetic wave, over a predetermined scanning range and receiving the returned component, and generates a distance measurement image based on the acquired distance measurement data. In particular, in this embodiment, the generated distance measurement image is corrected as necessary. Hereinafter, the distance image refers to all image data, including distance information obtained by performing corrections to the distance measurement image as well as distance measurement images.

[0020] Of the distance image device 100, the distance measurement unit 10 is composed of an optical unit 20 for performing optical scanning to obtain distance measurement data for generating a distance image, and a control unit 30 for controlling the optical scanning of the optical unit 20 as well as various operations for distance measurement.

[0021] In addition, the image processing unit 50 is composed of, for example, various circuit elements, a CPU, and a program (image processing program) for performing various processing in order to perform the necessary image processing on the ranging data or ranging image acquired by the ranging unit 10, and here includes a ranging data analysis unit 51, an abnormality judgment unit 52, and a correction unit 53.

[0022] In the distance measuring unit 10, the optical unit 20 includes a light projecting unit 21, a light scanning unit 22, and a light receiving unit .

[0023] The light projecting unit 21 is a laser light projecting unit that includes, for example, a laser light source, a lens, and the like, and that emits and projects laser light (pulsed light) in accordance with a command from the control unit 30.

[0024] The optical scanning unit 22 is a device for scanning a predetermined scanning range including, for example, the monitoring object OB to be used as the target TG with the laser light from the light projecting unit 21 in synchronization with the light projecting unit 21 in accordance with a command from the control unit 30. The optical scanning unit 22 can be configured with, for example, a two-dimensional scanning mirror (scanner) or the like, and scans light toward the monitoring object OB. In this embodiment, as an example, a Lissajous scanning type is adopted (see FIG. 2).

[0025] The light receiving unit 23 is a laser light receiving unit that receives the reflected component of the emitted laser light, and receives the reflected light of the scanned light by the optical scanning unit 22, i.e., the reflected component of the laser light. The light receiving unit 23 includes, in addition to a light receiving element (photodiode), for example, a light receiving optical system, a preamplifier, an A / D converter, etc. as necessary, and converts the reflected component into, for example, a detectable pulse wave state and outputs it to the control unit 30.

[0026] The control unit 30 includes a distance measurement control unit 31 and a scanning control unit 32 to control the various operations of the optical system components constituting the optical unit 20. The distance measurement control unit 31 performs overall control of the entire optical unit 20. Specifically, it outputs a command signal to the light projecting unit 21 to control the timing of laser light projection, and outputs various command signals to the scanning control unit 32 to cause Lissajous scanning synchronized with the light projection timing. The control unit 30 also receives information on the light reception results from the light receiving unit 23. As described above, the distance measurement control unit 31 obtains the time (time difference) from the projection of the laser light to the reception of its reflected component, or the distance value calculated based on this and data on the projection direction, as distance measurement data.

[0027] The scanning control unit 32 outputs various drive signals to the optical scanning unit 22 to drive the optical scanning unit 22 in accordance with the command signal from the distance measurement control unit 31. As described above, the scanning control unit 32 is of a Lissajous scanning type, and the optical scanning unit 22 moves in a sinusoidal wave manner in both the horizontal and vertical directions (double pendulum). More specifically, the laser light is irradiated in a manner that performs two-dimensional scanning to form a pattern figure as shown in FIG. 2, for example.

[0028] Returning to FIG. 1, the optical unit 20 further includes an abnormality detection unit 33, a data processing unit 34, and an IMU (Inertial Measurement Unit) 35.

[0029] The abnormality detection unit 33 detects abnormalities in the data acquired by the distance measurement control unit 31. In particular, here, it functions as an inversion detection unit RD that detects a time difference inversion abnormality in which the timing of receiving the reflected component at the light receiving unit 23 is detected as being earlier than the timing of projecting light by the light projecting unit 21 due to the influence of noise. Note that a typical example of the occurrence of a time difference inversion abnormality will be described later.

[0030] The data processing unit 34 performs processing on the data detected by the abnormality detection unit 33. In this example, the data processing unit 34 performs processing to treat the observation point (pixel) where the above-mentioned abnormality has occurred, that is, the observation point (pixel) where the time difference between the light projection timing and the light reception timing is detected as a negative value, as unusable.

[0031] The IMU 35 is a device for detecting the attitude, and detects, for example, the change in attitude of the optical scanning unit 22 during scanning.

[0032] The processing results of the data processing unit 34 and the attitude detection results of the IMU 35 are output to the communication control unit 40 together with the ranging data acquired by the scan control unit 32, and the communication control unit 40 outputs these data as ranging data to the image processing unit 50. The image processing unit 50 is capable of analyzing data from the ranging data transmitted from the image processing unit 50, for example, in units of one scan by Lissajous scanning, that is, one frame.

[0033] In addition to the above, the control unit 30 is provided with a power supply unit PW, which supplies the power necessary for the operation of each of the above-mentioned units.

[0034] On the other hand, as already described, the image processing unit 50 of the range imaging device 100 is provided with a ranging data analysis unit 51, an abnormality determination unit 52, and a correction unit 53. It is assumed that these units are configured, for example, such that the image processing unit 50 is composed of a CPU and a storage device, and performs necessary processing based on various programs stored in advance.

[0035] Of the image processing unit 50, the ranging data analysis unit 51 performs the necessary data processing and analysis on the ranging image (ranging data) based on the optical scanning by the optical scanning unit 22 and the light receiving results by the light receiving unit 23, so that it can be handled on a pixel-by-pixel basis.

[0036] The abnormality determination unit 52 determines whether or not there is an abnormality in the content of the ranging image (ranging data), i.e., the measurement results, and also performs various processes such as selecting target pixels required for this determination and preparing for the determination. Here, the presence or absence of an abnormality in a target pixel in the ranging image is determined based on the distance value of the target pixel, and the determination is made based on a comparison of the distance value of the target pixel with the distance values ​​of adjacent pixels adjacent to the target pixel. Furthermore, since this embodiment uses a Lissajous scan type, the adjacent pixels can include pixels on a scanning path different from the scanning path of the target pixel.

[0037] In particular, in a location where an obvious abnormality such as the time difference inversion described above has occurred, it is possible that distance measurement has not been performed properly not only at that location but also in the surrounding area, resulting in the calculation of an abnormal value. Therefore, in this embodiment, pixels where such an abnormality may have occurred are treated as determination target pixels in the abnormality determination unit 52. Note that an example of the selection of determination target pixels will be described later.

[0038] The correction unit 53 corrects the distance measurement image based on the result of the determination by the abnormality determination unit 52. That is, the correction unit 53 determines how to handle pixels that are determined to be abnormal as a result of the determination. Here, as an example, it is assumed that the pixel to be determined to be abnormal is treated as being unmeasurable, i.e., at an infinite distance where no reflected component is detected.

[0039] In this manner, the image processing unit 50 performs the necessary correction processing on the ranging data or ranging image acquired by the ranging unit 10, and generates a new ranging image (distance image) in which the distance information of the target ranging image has been corrected.

[0040] Problems with distance measurement in this embodiment and how to deal with them will be described below with reference to the conceptual diagram shown in Fig. 3. Fig. 3(A) is a conceptual diagram for explaining how distance measurement works under normal conditions, and Fig. 3(B) and Fig. 3(C) are conceptual diagrams for explaining how distance measurement works under abnormal conditions. Of these, Fig. 3(B) shows a typical example where a time difference inversion abnormality occurs, i.e., where the distance value becomes negative (an example where the occurrence of an obvious abnormality is evident), and Fig. 3(C) shows a typical example where a time difference shorter than the time difference that should have been detected is detected, resulting in a calculated distance value that is shorter than the actual distance value (occurrence of short-distance noise due to sunlight).

[0041] First, as shown in the conceptual diagram α1 in FIG. 3(A), in the case where the light LL from the light projecting unit 21 hits the target TG (monitoring object OB) and bounces back, the light receiving unit 23 receives the reflected component RR, the time difference Td from light projection to light reception is calculated as shown in graph α2. Specifically, in graph α2, the horizontal axis represents time (time) and the vertical axis represents the detected signal strength, with the upper part showing a curve LE indicating the strength of the light-emitting signal (pulsed light) on the light projecting side, and the lower part showing a curve LR indicating the strength of the light-receiving signal on the light receiving side. Detection thresholds are set in advance on both the light projecting side and the light receiving side, and the light projection timing and light reception timing are determined when the signal strength of the light-emitting signal and the light-receiving signal exceed the respective thresholds. In FIG. 3(A), time Te is the light projection timing and time Tr is the light reception timing. In this case, the time difference Td is calculated as follows: Td = Tr - Te (> 0) This becomes:

[0042] However, there may be cases where the above-described normal detection is not performed and an abnormal value is detected during distance measurement.

[0043] Generally, abnormal values ​​are detected during distance measurement when the light receiving unit 23 receives unintended light (disturbance light) and interprets it as a reflected component of the projected laser light. A typical example of such a situation is when light from the sun SU is received as disturbance light, as shown in Figures 3(B) and 3(C). For example, the laser light projected from the light projecting unit 21 may be one with a narrow wavelength band to avoid being affected by other components. However, the sun SU contains a certain amount of light in a very wide wavelength band. Therefore, even if a narrow wavelength band is used as the scanning light, the above-mentioned problems may not be avoided in an environment where components from the sun SU are present.

[0044] For example, as shown in the conceptual diagram β1 in Figure 3(B), when the direction of light projection and light reception is toward the sun SU (or, from another perspective, when the target TG is positioned with the sun at its back), there is a possibility that the component light SS from the sun SU will be received strongly from a point in time before the light projection timing. In this case, as shown by the curve LR on the light reception side in the graph β2, the value on the light reception side rises early, and the time Tr indicating the light reception timing may reach the detection threshold earlier than the time Te indicating the light projection timing. In other words, in this case, the inverted time difference Ti is Ti = Tr - Te (<0) This phenomenon can occur not only when sunlight directly enters the room as shown in the conceptual diagram β1, but also when the surface of a puddle on the floor reflects the component light SS of the sun SU.

[0045] Furthermore, as shown in the conceptual diagram γ1 in Figure 3(C), in cases where there is an influence not only of the reflected component RR from the target TG but also of the component light SS from the sun SU (for example, when the influence of the offset component in the detection at the previous pixel remains depending on the scanning direction), as shown by the curve LR on the light receiving side in graph γ2, the component of the received signal of the reflected component RR is added before the value has dropped completely due to the influence of the component light SS, and the detection threshold is reached at a stage earlier than the reflected component RR actually arrives. In other words, in this case, the value of the time difference Ts is compared with the value that should be detected (here, this is the time difference Td in graph α2), Ts <Td It will be detected in this state.

[0046] While values ​​like those in Figure 3(B) are clearly abnormal and can be excluded, values ​​like those in Figure 3(C) cannot necessarily be immediately recognized as abnormal, and it is desirable to find such situations and exclude or correct them as much as possible. By making it possible to respond in the manner described above, it becomes possible to avoid situations where, for example, when detecting whether or not there is a foreign object other than a normal installation, a difference in the distance measurement results for a normal installation causes the object to be recognized as a foreign object.

[0047] FIG. 4 illustrates an example of how the above-described situation occurs. Specifically, FIG. 4(A) is a conceptual image diagram illustrating an example of a target TG, FIG. 4(B) is a conceptual image diagram illustrating a distance image of the target TG and its surroundings under normal conditions when the range of the monitoring target is set to FIG. 4(A), and FIG. 4(C) is a conceptual image diagram illustrating a distance image of the target TG and its surroundings under abnormal conditions. In FIGS. 4(B) and 4(C), as an example, a distance measurement image (distance image) is displayed using a multi-level gradation for each distance value measured at each pixel position. Among these, locations where no reception was detected are recognized as the farthest and are displayed in black on the image. Furthermore, locations where clearly abnormal values ​​were measured, as illustrated in FIG. 3(B), are displayed in white on the image. Hereinafter, such obviously abnormal pixels displayed in white will be referred to as invalid pixels or distance measurement abnormal pixels.

[0048] First, in the example shown in FIG. 4(A), as shown in image GG, the imaging range as the range of the monitored object includes a floor surface FL as one of the targets TG, two monitored objects OB1 and OB2 (targets TG) fixedly (steadily) installed on the floor surface FL, and a space SP, such as the sky, as a background portion that should be captured as infinity. Note that monitored object OB1 is located relatively closer to the imaging position than monitored object OB2. Note that the floor surface FL gradually becomes farther away, i.e., it has depth. The space SP is assumed to be sufficiently far from the imaging position so that reflected components are not detected. An example of the monitored location shown in image GG is a train station platform. In other words, the floor of the station platform, platform doors, and the area where trains and passengers board and disembark are included in the range of the monitored object.

[0049] Under such circumstances, as shown in Figure 4(B), under normal conditions, the floor surface FL is displayed with a gradation pattern according to the distance as the distance increases, and the monitored objects OB1 and OB2 installed in predetermined positions are displayed with a single pattern according to the distance. On the other hand, the space SP is displayed in black as no light is detected.

[0050] However, as mentioned above, when ambient light (especially sunlight) is present, time difference inversion and distance value anomalies (measured distances shorter than the actual distances) can occur, as shown in FIG. 4C. As shown in the partially enlarged view A2 in the figure, pixels with time difference inversion are displayed as white pixels. As mentioned above, if the time difference is inverted, the time difference value is detected as a negative value, so the anomaly is immediately and clearly identified and can be treated as an invalid value. In contrast, as shown in another partially enlarged view A1, in a location corresponding to a portion of the monitored object OB1 in FIG. 4A, what should be displayed as approximately the same distance, or the same pattern, as the surrounding pixels is displayed as a different pattern. In other words, a different distance value may be detected. In other words, even if a distance value is calculated as a positive value, it may not be correct when compared to the distance values ​​of the surrounding pixels.

[0051] In contrast, the distance image device 100 of this embodiment determines whether or not there is an abnormality in a target pixel in the distance measurement image DG based on a comparison between the distance value of the target pixel and the distance values ​​of pixels adjacent to the target pixel, making it possible to deal with pixels such as those shown in the partially enlarged view A1. In particular, this embodiment makes use of the fact that scanning by Lissajous scanning results in different scanning directions and measurement timings even for adjacent pixels, resulting in differences between the target pixel and its adjacent pixels.

[0052] The characteristics of scanning by Lissajous scanning will be described below with reference to the conceptual diagrams shown as examples in FIGS. 5(A) to 5(D).

[0053] In Figures 5(A) to 5(D), arrows AR1 to AR4 indicate the paths of Lissajous scans. Figures 5(A) to 5(D) all illustrate examples in which the paths include pixels that show abnormal values ​​compared with the surrounding distance values ​​shown in Figure 3(C) and adjacent locations (areas DD surrounded by dashed lines in the figures).

[0054] As explained with reference to FIG. 2, in the case of Lissajous scanning, the scanning path is a combination of vertical and horizontal vibrations relative to a two-dimensional plane, as shown in FIGS. 5(A) to 5(D). For example, the arrow AR1 in FIG. 5(A) shows a top-to-bottom movement, while the horizontal movement changes from right-to-left to left-to-right. The arrow AR2 in FIG. 5(B) shows a left-to-right movement while moving from top-to-bottom. On the other hand, the arrow AR3 in FIG. 5(C) shows a bottom-to-top movement, while the horizontal movement changes from right-to-left to left-to-right. The arrow AR4 in FIG. 5(D) shows a right-to-left movement while moving from bottom-to-top. Of the above, FIGS. 5(A) and 5(D) show routes that pass through the space SP side, i.e., near the sun, and are therefore likely to be significantly affected by sunlight. On the other hand, in Figures 5(B) and 5(C), the route is from the monitored object OB1 etc. to the sun side (space SP side), and it is thought that the influence of sunlight is small. As such, in the case of scanning by Lissajous scanning, even distance measurement results for adjacent pixels have the characteristic that the degree of influence of sunlight (ambient light) may differ significantly depending on the route. In other words, even if a pixel adjacent to a distance measurement pixel that may have an abnormal distance value is measured, the measurement is performed on a different route, so there is a high possibility that normal pixels exist around the pixel with the abnormal measurement.

[0055] In consideration of the above characteristics, when determining whether or not there is an abnormality in a pixel to be determined that may have an abnormality, adjacent pixels to the pixel to be determined, for example, eight pixels surrounding the pixel to be determined, are extracted and used as comparison targets.

[0056] A series of pixel-by-pixel anomaly determination processes will be described below with reference to FIG. 6 and other figures. FIG. 6 is a conceptual diagram illustrating an example of a part of the procedure for identifying determination target pixels, and FIGS. 7(A) to 7(C) are conceptual diagrams illustrating an example of a process for identifying invalid pixels (ranging abnormal pixels) from one frame of a ranging image. Furthermore, FIGS. 8(A) to 8(C) are conceptual diagrams illustrating an example of a method for identifying determination target pixels based on invalid pixels (ranging abnormal pixels) and performing anomaly detection (anomaly determination) on the identified determination target pixels. Note that the series of anomaly determination processes described here are performed by anomaly determination unit 52 (see FIG. 1) of image processing unit 50. Furthermore, analysis of ranging data (ranging image) required for anomaly determination is performed by ranging data analysis unit 51.

[0057] Here, the processing procedure will be described in order with reference to Fig. 6 and subsequent figures. First, as shown in Fig. 6, as a premise, multiple distance measurement images are accumulated for identification using not only the target distance measurement data but also past distance measurement data. More specifically, for example, pixels that should be invalidated, i.e., invalid pixels that should be displayed as white (distance measurement abnormal pixels), are identified on a pixel-by-pixel basis for each frame acquired by Lissajous scanning, and then a distance measurement image arranged in a matrix is ​​created. This data is continuously acquired over a predetermined period (e.g., one second) to prepare multiple frames of data.

[0058] Then, as shown in Fig. 7, data from a predetermined time (e.g., the above-mentioned one second) from the most recent frame is used to trace back to past information to determine whether or not an invalid pixel (ranging error pixel) existed for each pixel position. At this time, the pixel at the position where the invalid pixel (ranging error pixel) existed within the predetermined time is identified (position marked with a star in the figure). After that, as shown in Fig. 8, pixels within a range expanded by a predetermined number of pixels (e.g., three pixels) from that pixel position are treated as judgment target pixels JP, and an abnormality judgment is performed.

[0059] The following provides further details. First, as described above, distance value calculation processing is sequentially performed on Lissajous scan-type distance measurement data corresponding to a predetermined number of frames. In the case of Lissajous scan, the matrix arrangement order and the measurement order differ. However, by sorting the distance measurement data for each frame, for example, the distance values ​​can be confirmed (checked) in the matrix arrangement order. In this manner, multiple distance measurement images DG are formed based on the data conceptually shown in FIG. 6. In other words, the presence or absence of invalid pixels and their positions are identified for multiple distance measurement images DG acquired continuously over a predetermined period. The illustrated example shows a situation in which one second's worth of distance measurement data is accumulated at 16 fps. In other words, 16 distance measurement images DG are prepared, including the current (most recent) distance measurement image DGα, in which the target pixel is to be identified.

[0060] 7A to 7C, a first stage of processing for identifying a target pixel in one frame of a distance measurement image DG will be described. A check is made to see if there is a time difference inversion anomaly in the past or current distance measurement image DG.

[0061] For example, as shown in Fig. 7(A), the pixels arranged in a matrix are checked in the horizontal direction (direction A) starting from the top left pixel PX, and then as shown in Fig. 7(B), the check proceeds in the vertical direction (direction B) sequentially downward. If a time difference inversion anomaly is found in the past or present as a result of the check, the pixel PX is indicated by a black star as shown in Fig. 7(C) as an example. Based on the check result, it is determined whether or not to determine whether there is an anomaly in the current ranging image DGα (which part of the ranging image DGα should be the target of the determination).

[0062] Next, with reference to the conceptual diagrams shown in Figures 8(A) to 8(C), an example of a method for identifying a target pixel JP in a ranging image DGα and performing abnormality detection (abnormality determination) on the identified target pixel JP will be described.

[0063] First, FIG. 8A is a conceptual diagram of a portion of the pixels constituting the ranging image DGα. Here, a total of 49 pixels PX—7 in the A direction (horizontal direction) and 7 in the B direction (vertical direction)—are located at coordinates A1-A7 and B1-B7. In particular, pixel PXc, located at coordinates (A4, B4) in the center, is assumed to be the pixel where the time difference inversion anomaly described with reference to FIG. 7 occurred. This is indicated by a black star in the figure. In this example, as indicated by arrows AA1-AA8, pixels within three pixels of pixel PXc, i.e., the pixel where the time difference inversion anomaly occurred, are designated as target pixels JP. Therefore, in this case, all 49 pixels PX in the figure are treated as target pixels JP. This is because, as described with reference to FIG. 3, in areas where a clear anomaly such as a time difference inversion anomaly has occurred in the past or present, i.e., where a situation like that shown in FIG. 3B occurred, there is a high possibility that some kind of disturbance light has also been present in the vicinity of that location. For example, it is highly likely that a situation like that shown in FIG. 3C has occurred. Therefore, in this embodiment, the area around the location where the time difference inversion anomaly occurred is identified as described above, and further anomaly determination is performed within this area.

[0064] An example of a determination method for one determination target pixel will be described below. For example, for one pixel PX1 located at coordinates (A5, B2) shown in FIG. 8B, the eight pixels surrounding it are set as adjacent pixels NP adjacent to the determination target pixel PX1, as shown in FIG. 8C. The distance value of the determination target pixel PX1 is then compared with the distance value of the adjacent pixels NP, and if there is a difference greater than a predetermined standard (for example, if the difference in distance is ±15% or more), it is determined that there is an abnormality in the measurement result of the determination target pixel PX1. This process is performed for pixel PXc and the 49 pixels (determination target pixels) PX surrounding it.

[0065] As a result of the abnormality determination process described above, it is assumed that pixels that are determined to be abnormal will undergo correction processing, such as treating them as pixels that display black (no reception). Note that such image correction is performed by the correction unit 53 of the image processing unit 50 (see FIG. 1).

[0066] A series of processes for correcting a distance image (range measurement image) will be described below with reference to the flowcharts shown in FIGS.

[0067] First, as shown in Fig. 9, in process S1, surrounded by a dashed line, the content of distance measurement in one frame of a distance measurement image obtained by Lissajous scanning is analyzed. Specifically, one frame of distance measurement data, i.e., all pixels, is extracted (step S101). For each pixel, the presence or absence of a light reception signal is first confirmed (step S102). If no signal is detected (step S102: No), the pixel is treated as having been unable to be measured (step S103). In other words, the pixel is treated as having no reception detection and should be displayed in black (infinite distance).

[0068] On the other hand, if a received light signal is confirmed in step S102 (step S102: Yes), it is checked whether the time difference between transmission and reception is a positive value (step S104). If it is not confirmed (step S104: No), that is, if the time difference is zero or a negative value, it is treated as having a distance measurement abnormality (step S105). This corresponds to the situation illustrated in Fig. 3(B), and the pixel in question is treated as one that should be displayed in white (one that should be an invalid pixel or a distance measurement abnormality pixel) as illustrated in Fig. 4(C).

[0069] On the other hand, if it is confirmed in step S104 that the time difference is a positive value (step S104: Yes), a distance value based on the time difference is calculated (step S106).

[0070] In this way, pixels are classified into either pixels for which distance calculation has been performed in step S106, pixels that are displayed as black (infinite distance), or pixels that are displayed as white (invalid pixels or distance measurement abnormal pixels), and this is repeated until it is performed for all pixels in one frame (step S107).

[0071] When the classification process for all pixels shown in step S1 is completed, the distance measurement data for one frame obtained by the Lissajous scan is arranged in a matrix (two-dimensional array). That is, the data is sorted so that it can be displayed as a distance image after distance value calculation processing for the distance measurement image (step S108). Note that by sequentially performing step S1 for each frame, i.e., by repeating step S1, data related to distance values ​​on a pixel-by-pixel basis is accumulated for multiple continuously acquired distance measurement images (distance images).

[0072] Next, as shown in Fig. 10, a two-dimensionally arranged distance image (distance values ​​calculated pixel by pixel for the ranging image) is subjected to a judgment as to whether the calculated distance values ​​in the frame are abnormal, as illustrated in Fig. 6 and Fig. 7. In process S2 shown in Fig. 10, a preprocessing step (processing equivalent to the step of identifying the star illustrated in Fig. 7) is performed to select pixels to be judged from among the pixels constituting the current frame (corresponding to ranging image DGα in Fig. 6), which is the most recent distance image for which an abnormality judgment regarding the distance values ​​is to be made.

[0073] First, the number of vertical pixels (the number of pixels in the vertical direction) for the current frame is extracted (step S201). Further, the number of horizontal pixels (the number of pixels in the horizontal direction) is extracted for each vertical value (corresponding to the coordinate value in the B direction in FIG. 7) (step S202). A check is then performed on each pixel in the horizontal direction one by one (steps S202 to S205). Specifically, for each pixel, it is first determined whether there is a ranging anomaly, i.e., a time difference inversion anomaly, at that pixel's position in the current frame and the frames up to one second in the past (a total of 16 frames for 16 fps) (step S203). If there is a ranging anomaly in the current or past frame in step S203 (step S203: Yes), an anomaly flag is set for that pixel (step S204). On the other hand, if there is no ranging anomaly in step S203 (step S203: No), no special processing is performed.

[0074] The above process is repeated until there are no more pixels in the horizontal direction, i.e., until all pixels in a horizontal row have been processed (step S205). When the horizontal row has been processed, the process advances vertically by one pixel, and the above process is performed again in the horizontal direction, until all pixels constituting the current frame have been processed (step S206).

[0075] In this way, pre-processing for selecting pixels to be judged, that is, processing for determining whether or not a pixel should be marked with a star (a pixel at a position where a time difference inversion anomaly occurred) as shown in Fig. 7, is performed. The pre-processing for all pixels shown in process S2 is then completed.

[0076] Next, as shown in process S3 in Fig. 11, to identify the target pixel, the process of expanding the target as illustrated in Fig. 8(A) is performed, that is, a process of treating pixels within a range expanded by a predetermined number of pixels from the position where the time difference inversion anomaly occurred between the past one second and the present as the target pixel. Note that in the example in Fig. 8(A), the target pixel is one within three pixels from the pixel where the time difference inversion anomaly occurred. In other words, the range is expanded three times. However, the range to be expanded is not limited to this and can be various, for example, it may be expanded two times.

[0077] Here, a check flag is set for a pixel to be determined. Specifically, first, for the current frame, the number of vertical pixels (the number of pixels in the vertical direction) is extracted (step S301), and then the number of horizontal pixels (the number of pixels in the horizontal direction) is extracted for each vertical value (corresponding to the coordinate value in the B direction in FIG. 8) (step S302), and each pixel in the horizontal direction is checked one by one.

[0078] To explain the above confirmation in more detail, first, it is confirmed whether the target pixel is flagged as abnormal (whether a time difference inversion anomaly has occurred between the past 1 second and the present) (step S303), and if it is determined in step S303 that the target pixel is a ranging anomaly (time difference inversion anomaly) (step S303: Yes), check flags are set for the pixel and its surrounding 48 pixels (pixels enlarged three times), i.e., a total of 49 pixels, as pixels to be determined (check flags are assigned) (step S304). Note that if it is confirmed in step S303 that no abnormality flag is set, i.e., that there is no ranging anomaly (time difference inversion anomaly), (step S303: No), no special processing is performed on the pixel or its surrounding pixels.

[0079] The above process is repeated until there are no more pixels in the horizontal direction, that is, until it has been performed on all pixels in one horizontal row (step S305).When one horizontal row is completed, the process advances vertically by one pixel, and the above process is performed again in the horizontal direction, until it has been performed on all pixels that make up the current frame (step S306).

[0080] Finally, as shown in Fig. 12, a series of processes is completed through processes S4 to S6 for all pixels constituting the current frame, including the pixel to be determined (those with a check flag attached). As a prerequisite for performing processes S4 to S6, the number of vertical pixels (the number of pixels in the vertical direction) is extracted for all pixels constituting the current frame (step S401), and then the number of horizontal pixels (the number of pixels in the horizontal direction) is extracted for each vertical axis (step S402), and confirmation is performed for each pixel in the horizontal direction one by one.

[0081] First, in process S4, pixels with distance measurement abnormalities (those with time difference inversion and displayed in white) are excluded (step S403w), and pixels with distance unmeasurable (those with infinite distance and displayed in black) are excluded (step S403b). Only pixels that are determined to be neither of these (No in both step S403w and step S403b) are further judged to have a check flag attached (step S404). Only pixels with a check flag present in step S404 (step S404: Yes), i.e., pixels determined to be pixels to be judged, proceed to process S5 and subsequent steps. In other words, pixels with distance measurement abnormalities or distance unmeasurable (Yes in either step S403w or step S403b) and pixels with no check flag present in step S404 (step S404: No) are not further processed.

[0082] Next, in step S5, the distance values ​​are compared between the target pixel and its eight neighboring pixels. That is, the process described with reference to FIGS. 8B and 8C is performed. More specifically, the distance values ​​for the eight neighboring pixels are extracted (step S405). The distance values ​​of each neighboring pixel are compared with the distance value of the target pixel to determine whether the difference is within ±15% (step S406). If the difference is within ±15% (step S406: Yes), the match count is incremented by one to treat the distance values ​​as matching (step S407). On the other hand, if the difference is not within ±15% (i.e., the distance values ​​do not match) (step S406: No), no special process is performed (the match count remains unchanged). The above process is continued until all eight neighboring pixels have been processed (step S408).

[0083] Next, in step S6, it is determined whether the target pixel being processed should be corrected, i.e., whether it should be treated as an unmeasurable pixel (displayed in black with an infinite distance). More specifically, based on the result of the match count performed in step S407, it is checked whether the number of adjacent pixels with matching distance values ​​is equal to or less than a predetermined threshold (step S409). If it is equal to or less than the threshold (step S409: Yes), the target pixel is treated as an unmeasurable pixel (displayed in black) due to an abnormality in the distance measurement result (step S410). On the other hand, if it is determined in step S409 that it is not equal to or less than the threshold (step S409: No), no special processing is performed. In other words, the target pixel is treated as if no abnormality was confirmed in the calculated distance value.

[0084] The above processes S4 to S6 are repeated until there are no more pixels in the horizontal direction, i.e., until they have been performed on all pixels in one horizontal row (step S411).When one horizontal row has been completed, the process advances vertically by one pixel, and the above processes are performed again in the horizontal direction, until they have been performed on all pixels to be determined (step S412).

[0085] Hereinafter, the overall configuration of the process for determining abnormality in a distance image by the distance imaging device 100 will be outlined with reference to the conceptual diagram shown in FIG.

[0086] As shown in the figure and as described above, the range image device 100 includes an optical scanning unit 22, a light receiving unit 23, and an abnormality determination unit 52. The optical scanning unit 22 scans light (laser light) toward the monitoring target OB. That is, the light (laser light) toward the monitoring target OB becomes scanned light. The light receiving unit 23 receives reflected light of the scanned light (laser light) by the optical scanning unit 22. Based on the above-described scanned light and reception of its reflected light (reflected component), i.e., based on the optical scanning by the optical scanning unit 22 and the light reception results by the light receiving unit 23, the range image device 100 is capable of acquiring data for forming a range image (or distance image) DG. That is, based on the optical scanning by the optical scanning unit 22 and the light reception results by the light receiving unit 23, the range image device 100 calculates distance values ​​on a pixel-by-pixel basis (distance value calculation process). Then, the abnormality judgment unit 52 determines a target pixel JP from the acquired distance measurement image DG that may exhibit an abnormal value, compares the distance value of the target pixel JP with the distance value of the adjacent pixel NP adjacent to the target pixel JP to determine whether or not there is an abnormality in the determined target pixel JP (distance value comparison process), and makes a judgment based on the comparison result.The correction unit 53 then corrects the distance value on a pixel-by-pixel basis in accordance with the judgment result (distance measurement value correction process).

[0087] As described above, the distance image device 100 of this embodiment includes the optical scanning unit 22 that scans light toward the monitoring object OB, the light receiving unit 23 that receives reflected light of the scanned light by the optical scanning unit 22, and the abnormality determination unit 52 that determines the presence or absence of an abnormality in the determination target pixel JP in the distance image DG based on the optical scanning by the optical scanning unit 22 and the light receiving result at the light receiving unit 23, based on a comparison between the distance value of the determination target pixel JP and the distance value of the neighboring pixel NP adjacent to the determination target pixel JP. In this case, the distance image device 100 can accurately detect abnormalities on a pixel-by-pixel basis in the distance image (distance image) DG by determining the presence or absence of an abnormality based on a comparison between the distance value of the determination target pixel JP and the distance value of the neighboring pixel NP adjacent to the determination target pixel JP.

[0088] 〔others〕 The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit and scope of the present invention.

[0089] First, the method described above is an example, and the threshold value to be set, the degree of difference to be compared, the target range, etc. can be changed depending on the installation environment, the accuracy required for the purpose of monitoring, etc. For example, in the above example, the range to be expanded three times when determining the target pixel is expanded, but as already mentioned, this can also be expanded twice, for example, and in this case, a total of 25 pixels, consisting of one pixel with an abnormal flag and its surrounding 24 pixels (pixels expanded twice), become the target pixels.

[0090] Furthermore, in the above description, pixels with abnormal measurement values ​​are treated as black (infinity), but other correction methods may be used (for example, taking into account surrounding values ​​and using a value stored based on these).

[0091] Furthermore, various types of light sources and light receiving elements for optical scanning can be used, and for example, an avalanche photodiode (APD) can be used. [Explanation of symbols]

[0092] 10... ranging unit, 20... optical unit, 21... light projecting unit, 22... optical scanning unit, 23... light receiving unit, 30... control unit, 31... ranging control unit, 32... scanning control unit, 33... abnormality detection unit, 34... data processing unit, 35... inertial measurement unit (IMU), 40... communication control unit, 50... image processing unit, 51... ranging data analysis unit, 52... abnormality determination unit, 53... correction unit, 100... distance image device, A1 to A7, B1 to B7... coordinates, AA1 to AA8... arrows, AR1 to AR4... arrows, DD... area, D G... ranging image (distance image), DGα... ranging image, FL... floor, GG... image, JP... pixel to be judged, LE... curve, LL... light, LR... curve, NP... adjacent pixel, OB, OB1, OB2... monitoring target, PW... power supply unit, PX, PX1, PXc... pixel, RD... inversion detection unit, RR... reflected component, SP... space, SS... component light, SU... sun, TG... target, Td, Ti, Ts... time difference, Te, Tr... time, α1, β1, γ1... conceptual diagram, α2, β2, γ2... graph

Claims

1. a light scanning unit that scans light toward a monitoring target; a light receiving unit that receives reflected light of the scanning light from the optical scanning unit; an abnormality determination unit that determines whether or not there is an abnormality in a determination target pixel in a distance measurement image based on the optical scanning by the optical scanning unit and the light receiving result by the light receiving unit, based on a comparison between a distance value of the determination target pixel and a distance value of an adjacent pixel adjacent to the determination target pixel; A distance imaging device comprising:

2. the optical scanning unit is a Lissajous scanning type, The distance imaging device according to claim 1 , wherein the abnormality determination unit includes, as the neighboring pixels, pixels on a scanning path different from a scanning path of the determination target pixel.

3. The distance imaging device according to claim 1 , wherein the abnormality determination unit treats eight pixels surrounding the determination target pixel as the adjacent pixels.

4. 4. The distance imaging device according to claim 3, wherein the abnormality determination unit determines that the target pixel has an abnormality when there is a distance difference between the target pixel and a predetermined number of the eight pixels or more.

5. 3. The distance imaging device according to claim 1, wherein a determination is made as to whether or not an abnormality exists in the current distance measurement image based on whether or not a time difference inversion abnormality exists in past or current distance measurement images between multiple distance measurement images acquired continuously during a predetermined period.

6. 6. The distance image device according to claim 5, wherein, when the time difference inversion anomaly is detected, the anomaly determination unit treats pixels in the current ranging image that are within a range expanded by a predetermined number of pixels from the position where the time difference inversion anomaly occurred as the target pixels for determination.

7. The distance imaging device according to claim 1 , further comprising a correction unit that corrects the distance value in pixel units in accordance with the determination result of the abnormality determination unit.

8. a distance value calculation step of calculating a distance value in pixel units based on the optical scanning by the optical scanning unit and the light receiving result by the light receiving unit; a distance value comparison step of comparing a distance value between a target pixel and an adjacent pixel adjacent to the target pixel in the distance measurement image generated by the distance value calculation step; a distance measurement value correction step of determining whether or not there is an abnormality in the distance measurement image based on the comparison result in the distance value comparison step, and correcting the distance value in pixel units in accordance with the determination result; A distance image correction method comprising:

9. a distance value calculation process for calculating a distance value in pixel units based on the optical scanning by the optical scanning unit and the light receiving result by the light receiving unit; a distance value comparison process for comparing the distance values ​​of a determination target pixel and adjacent pixels adjacent to the determination target pixel in the distance measurement image generated by the distance value calculation process; a distance measurement value correction process that determines whether or not there is an abnormality in the distance measurement image based on a comparison result in the distance value comparison process, and corrects the distance value on a pixel-by-pixel basis in accordance with the determination result; , image processing program.

Images