Image processing device and image processing method

Abstract

In this invention, an input interface (21) acquires a first distance image including a plurality of pixels indicating distance values from an imaging device (1) to various points on an object. An image division device (24) uses the distance values of each pixel to divide the first distance image into a plurality of pixel groups such that each of the plurality of pixel groups includes pixels having distance values included in one of a plurality of different distance intervals. A noise filter (25) reduces the noise of the plurality of pixel groups through individual processing of the plurality of pixel groups in which a plurality of different filter parameters are respectively used for the plurality of pixel groups. An image combination device (26) generates a second distance image by combining the plurality of pixel groups processed by the noise filter (25).

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] 2. Description of the Related Art Imaging devices such as a ToF (Time of Flight) camera are known that acquire distance images including a plurality of pixels each indicating a distance value to each point on an object.

[0003] For example, Patent Document 1 discloses an object position detection device that outputs the position of an object such as a person as a distance image, and Patent Document 2 discloses a filter processing device that filters three-dimensional distance image data input from a three-dimensional sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6814053 [Patent Document 2] Patent No. 6793055 Summary of the Invention [Problem to be solved by the invention]

[0005] Range images contain random noise due to factors such as insufficient sensitivity of the image sensor and thermal noise in the image sensor and circuitry, and therefore, it is necessary to reduce this noise. Range images can be considered three-dimensional data having vertical, horizontal, and depth coordinates as viewed from the image sensor. However, because range images are technically two-dimensional data, two-dimensional image processing can be applied. If noise reduction methods used in conventional image processing, such as median filters, are applied to range images as is, there is a risk that the noise will be amplified or missing pixels will be enlarged. Therefore, it is necessary to reduce noise in range images more reliably than before.

[0006] The present disclosure provides an image processing device and an image processing method that can more reliably reduce noise in a distance image than conventional methods. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present disclosure includes: an input interface for acquiring a first distance image including a plurality of pixels each indicating a distance value from the imaging device to each point on the object; an image divider that divides the first distance image into a plurality of pixel groups based on the distance values ​​of each pixel, such that each pixel group includes a pixel having a distance value that is included in one of a plurality of distance intervals that are different from each other; a noise filter that processes the plurality of pixel groups individually using a plurality of filter parameters that are different from each other for each of the plurality of pixel groups to reduce noise in the plurality of pixel groups; and an image combiner that combines the plurality of pixel groups processed by the noise filter to generate a second distance image. [Effects of the Invention]

[0008] According to an image processing device according to an aspect of the present disclosure, noise in a distance image can be reduced more reliably than in the past. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing the configuration of an image processing device 2 according to a first embodiment. [Figure 2] 2 is a flowchart showing a noise reduction process executed by the processing circuit 20 of FIG. 1. [Figure 3] 2 is a diagram showing an exemplary distance image 40 processed by the image processing device 2 of FIG. 1. [Figure 4] FIG. 4 is a diagram showing a pixel group 40a corresponding to distance section D1 included in distance image 40 of FIG. [Figure 5]FIG. 4 is a diagram showing a pixel group 40b included in the distance image 40 of FIG. 3 and corresponding to distance section D2. [Figure 6] FIG. 4 is a diagram showing a pixel group 40c corresponding to distance section D3 included in distance image 40 of FIG. [Figure 7] FIG. 4 is a diagram showing a pixel group 40d that corresponds to distance section D4 and is included in distance image 40 of FIG. [Figure 8] FIG. 4 is a diagram showing a pixel group 40e that corresponds to distance section D5 and is included in distance image 40 of FIG. [Figure 9] 5 is a diagram showing a pixel group 40a' after the pixel group 40a in FIG. 4 has been processed by a noise filter 25. FIG. [Figure 10] 6 is a diagram showing a pixel group 40b' after the pixel group 40b of FIG. 5 has been processed by a noise filter 25. FIG. [Figure 11] 7 is a diagram showing a pixel group 40c' after the pixel group 40c of FIG. 6 has been processed by a noise filter 25. FIG. [Figure 12] 8 is a diagram showing a pixel group 40d' after the pixel group 40d in FIG. 7 has been processed by a noise filter 25. FIG. [Figure 13] 9 is a diagram showing a pixel group 40e' after the pixel group 40e of FIG. 8 has been processed by a noise filter 25. FIG. [Figure 14] FIG. 14 is a diagram showing a distance image 40' obtained by combining pixel groups 40a' to 40e' of FIGS. 9 to 13. [Figure 15] FIG. 10 is a schematic diagram showing the configuration of an image processing device 2A according to a second embodiment. [Figure 16] 16 is a flowchart showing a noise reduction process executed by the processing circuit 20A of FIG. 15. [Figure 17] FIG. 10 is a schematic diagram showing the configuration of an image processing device 2B according to a third embodiment. [Figure 18] FIG. 10 is a schematic diagram showing the configuration of an image processing device 2C according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed description than necessary may be omitted. For example, detailed description of well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art.

[0011] The inventor(s) provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0012] [First embodiment] [Configuration of the first embodiment] 1 is a schematic diagram showing the configuration of an image processing device 2 according to the first embodiment. The image processing device 2 acquires a distance image from an imaging device 1 and reduces noise, such as random noise, contained in the distance image.

[0013] The imaging device 1 generates a distance image including a plurality of pixels each indicating a distance value from the imaging device 1 to each point on an object. The imaging device 1 may be, for example, a ToF (Time of Flight) camera, a LIDAR (Light Detection and Ranging) camera, or a stereo camera.

[0014] The image processing device 2 includes a processing circuit 20, an input interface (I / F) 21, an output interface (I / F) 22, and a storage device 23. The processing circuit 20 includes an image divider 24, a noise filter 25, an image combiner 26, and a filter controller 27.

[0015] The input interface 21 acquires a distance image from the imaging device 1 and sends it to the image divider 24 of the processing circuit 20. The input interface 21 may be, for example, a signal interface such as a USB (Universal Serial Bus) or Ethernet (registered trademark).

[0016] The image divider 24 acquires a distance image from the imaging device 1 via the input interface 21 and divides the distance image into multiple pixel groups based on the distance value of each pixel. The image divider 24 divides the distance image into multiple pixel groups based on the distance value of each pixel, such that each pixel group includes pixels having distance values ​​that fall within one of multiple distance intervals that are different from one another. Each distance interval represents a division of the distance from the imaging device 1 to each point on the object. Each of the multiple pixel groups has a representative distance value that represents the distance values ​​of the pixels included in that pixel group. The representative distance value may be, for example, the minimum, maximum, or average distance value of the pixels included in the pixel group.

[0017] The noise filter 25 reduces noise in the pixel groups by individually processing the pixel groups using a plurality of filter parameters that are different from each other for each pixel group. The noise filter 25 includes, for example, one or more median filters. The noise filter 25 may process the pixel groups sequentially or may process the pixel groups in parallel.

[0018] The storage device 23 stores in advance a plurality of filter parameters for a plurality of pixel groups. The filter parameters are set, for example, so that the performance of the noise filter 25 in reducing noise decreases as the representative distance value of the pixel group increases. The higher the performance of the noise filter 25, the easier it is to reduce or remove noise, but the more likely it is that missing pixels will occur. The filter parameters include, for example, the filter window width.

[0019] The filter controller 27 reads out the filter parameters from the storage device 23 and sets them in the noise filter 25. The filter controller 27 sets the filter parameters in the noise filter 25 for each pixel group.

[0020] The image synthesizer 26 synthesizes the plurality of pixel groups processed by the noise filter 25 to generate a distance image.

[0021] The output interface 22 sends the distance image generated by the image synthesizer 26 to a downstream processing device (not shown). The output interface 22 may be, for example, a signal interface such as USB or Ethernet (registered trademark). The downstream processing device may include, for example, an image recognizer.

[0022] The processing circuitry 20 may include multiple dedicated circuits corresponding respectively to the image divider 24, the noise filter 25, the image combiner 26, and the filter controller 27. Alternatively, the processing circuitry 20 may include one or more general-purpose or dedicated processors (e.g., digital signal processors) that operate as the image divider 24, the noise filter 25, the image combiner 26, and the filter controller 27 by executing predetermined programs.

[0023] The image processing device 2 may process a still image to reduce noise therein, or may process a moving image to reduce noise therein.

[0024] [Operation of the first embodiment] FIG. 2 is a flowchart showing the noise reduction process performed by the processing circuit 20 of FIG.

[0025] The processing circuit 20 acquires a distance image from the imaging device 1 via the input interface 21 (step S1).

[0026] Next, the processing circuit 20 divides the distance image into a plurality of pixel groups based on the distance value of each pixel (step S2).

[0027] Steps S1 to S2 show the operation of the processing circuit 20 as the image divider 24.

[0028] Next, processing circuit 20 selects one pixel group from the plurality of pixel groups divided in step S2 (step S3).

[0029] Next, the processing circuit 20 selects and reads out from the storage device 23 the filter parameters corresponding to the pixel group selected in step S3, and sets them as the filter parameters of the noise filter 25 (step S4).

[0030] As mentioned above, the filter parameters include, for example, the filter window width. Each pixel group is represented by i=0, 1, ..., k, where i=0 is the pixel group closest to the image capture device 1 and i=k is the pixel group farthest from the image capture device 1. The window width h of each pixel group is i is defined, for example, by the following formula:

[0031] h i =h0-(a×i)+λg

[0032] Here, h0 indicates the initial value of the window width, a indicates the rate of decrease of the window width, g indicates the gain of the image capture device 1, and λ indicates the mixing ratio. i is set to always be greater than or equal to 0.

[0033] This window width h i According to the formula, as the pixel group moves away from the image capture device 1, the window width h i Therefore, the filter parameters are set such that the performance of the noise filter 25 decreases as the representative distance value of the pixel group increases.

[0034] Processing circuitry 20 then processes the pixel group to reduce noise in the pixel group in accordance with the set filter parameters (step S5).

[0035] Next, processing circuit 20 determines whether or not all pixel groups have been processed to reduce noise (step S6), and if YES, proceeds to step S8, and if NO, proceeds to step S7.

[0036] Next, processing circuit 20 selects the next unselected pixel group from the plurality of pixel groups divided in step S2 (step S7), and then repeats steps S4 to S6.

[0037] Steps S3 and S5 to S7 show the operation of the processing circuit 20 as the noise filter 25, and step S4 shows the operation of the processing circuit 20 as the filter controller 27.

[0038] Next, the processing circuit 20 generates a distance image by combining the plurality of pixel groups that have been processed to reduce noise (step S8).

[0039] Next, the processing circuit 20 outputs the synthesized distance image to a downstream processing device via the output interface 22 (step S9).

[0040] Steps S8 to S9 show the operation of the processing circuit 20 as the image synthesizer .

[0041] Next, exemplary noise reduction processing executed by the image processing device 2 of FIG. 1 will be described with reference to FIGS.

[0042] 3 is a diagram showing an exemplary distance image 40 processed by the image processing device 2 of FIG. 1. The distance image 40 includes objects 41-44, a ground 45, and a background 46. The objects 41-44 include, for example, people, cars, and trees. The background 46 is essentially a point at infinity, and its pixels have a distance value of zero (or no distance value). Furthermore, the distance image 40 includes noise 47. The noise 47 is a pixel or region having a distance value that is discontinuous from the distance values ​​of surrounding pixels due to insufficient sensitivity of the image sensor, thermal noise of the image sensor and circuitry, etc.

[0043] In Figure 3 and other figures, for the sake of explanation, the noise 47 is shown by large circles, squares, stars, triangles, and small circles, but in reality, most of the noise 47, which is random noise, is not a continuous area but is made up of isolated pixels.

[0044] As described above, the image divider 24 divides the distance image into a plurality of pixel groups based on the distance values ​​of each pixel, such that each of the plurality of pixel groups includes pixels having distance values ​​that fall within one of a plurality of mutually different distance intervals. In the example of Figures 3 to 14, the distance from the image capture device 1 to each point on the object is divided into five distance intervals D1 to D5. Here, for example, distance interval D1 is an interval that includes distance values ​​of 0 to 5 m, distance interval D2 is an interval that includes distance values ​​of 5 to 10 m, distance interval D3 is an interval that includes distance values ​​of 10 to 15 m, distance interval D4 is an interval that includes distance values ​​of 15 to 20 m, and distance interval D5 is an interval that includes distance values ​​of 20 m or more.

[0045] 4 is a diagram showing pixel group 40a corresponding to distance section D1 and included in distance image 40 of FIG. 3. Pixel group 40a includes ground 45a and noise 47a. Pixel group 40a also includes defective pixels 48a corresponding to pixels of the object, ground, or noise included in other pixel groups, i.e., pixels having distance values ​​not included in distance section D1. Because the pixels corresponding to defective pixel 48a are included in other pixel groups, the pixels of defective pixel 48a have a distance value of zero (or no distance value) in pixel group 40a.

[0046] Fig. 5 is a diagram showing a pixel group 40b corresponding to distance section D2, included in distance image 40 of Fig. 3. Pixel group 40b includes an object 41, a ground 45b, noise 47b, and a missing pixel 48b.

[0047] Fig. 6 is a diagram showing a pixel group 40c corresponding to distance section D3 included in distance image 40 of Fig. 3. Pixel group 40c includes an object 42, a ground surface 45c, noise 47c, and a missing pixel 48c.

[0048] Fig. 7 is a diagram showing a pixel group 40d corresponding to distance section D4, included in distance image 40 of Fig. 3. Pixel group 40d includes an object 43, a ground surface 45d, noise 47d, and a missing pixel 48d.

[0049] Fig. 8 is a diagram showing a pixel group 40e corresponding to distance section D5, included in distance image 40 of Fig. 3. Pixel group 40e includes an object 44, a ground 45e, noise 47e, and a missing pixel 48e.

[0050] The noise filter 25 processes the pixel groups 40a to 40e individually to reduce noise in the pixel groups 40a to 40e, as will be described below with reference to FIGS.

[0051] FIG. 9 shows pixel group 40a' after processing pixel group 40a in FIG. 4 with noise filter 25. Filtering corrects the distance values ​​of most of the pixels in noise 47a in FIG. 4 according to the pixel values ​​of their surrounding pixels, and the pixels in noise 47a become corrected pixels with corrected distance values. In the example of FIG. 9, if the pixels surrounding noise 47a have distance values ​​of zero, filtering changes the pixels in noise 47a to corrected pixels 49a-1 with a distance value of zero. Furthermore, if noise 47a occurs on ground 45a, filtering changes the pixels in noise 47a to corrected pixels 49a-2 with a distance value equal to the distance values ​​of the pixels on ground 45a. However, if multiple noises 47a are close to or densely packed together, filtering may make it difficult to correct the pixels in noise 47a and they may remain as they are.

[0052] FIG. 10 illustrates pixel group 40b' after processing pixel group 40b in FIG. 5 with noise filter 25. The distance values ​​of most of the pixels in noise 47b in FIG. 5 are corrected by filtering according to the pixel values ​​of their surrounding pixels, and the pixels in noise 47b become corrected pixels having corrected distance values. In the example of FIG. 10, if the pixels surrounding noise 47b have distance values ​​of zero, filtering results in the pixels in noise 47b becoming corrected pixels 49b-1 having a distance value of zero. If noise 47b occurs on ground 45b, filtering results in the pixels in noise 47b becoming corrected pixels 49b-2 having the same distance value as the pixels in ground 45b. If noise 47b occurs on object 41, filtering results in the pixels in noise 47b becoming corrected pixels 49b-3 having the same distance value as the pixels in object 41. However, if multiple noise elements 47b are close to or densely packed together, the pixels in noise 47b are unlikely to be corrected even after filtering, and may remain as they are.

[0053] FIG. 11 shows pixel group 40c' after processing pixel group 40c in FIG. 6 with noise filter 25. The distance values ​​of most of the pixels in noise 47c in FIG. 6 are corrected by filtering according to the pixel values ​​of their surrounding pixels, and the pixels in noise 47c become corrected pixels having corrected distance values. In the example of FIG. 11, if the pixels surrounding noise 47c have distance values ​​of zero, filtering results in the pixels in noise 47c becoming corrected pixels 49c-1 having a distance value of zero. If noise 47c occurs on ground 45c, filtering results in the pixels in noise 47c becoming corrected pixels 49c-2 having the same distance value as the pixels in ground 45c. If noise 47c occurs on object 42, filtering results in the pixels in noise 47c becoming corrected pixels 49c-3 having the same distance value as the pixels in object 42. However, if multiple noises 47c are close to or densely packed together, filtering may make it difficult to correct the pixels in noise 47c and they may remain as they are.

[0054] FIG. 12 shows pixel group 40d' after processing pixel group 40d in FIG. 7 with noise filter 25. The distance values ​​of most of the pixels in noise 47d in FIG. 7 are corrected by filtering according to the pixel values ​​of their surrounding pixels, and the pixels in noise 47d become corrected pixels having corrected distance values. In the example of FIG. 12, if the pixels surrounding noise 47d have distance values ​​of zero, filtering results in the pixels in noise 47d becoming corrected pixels 49d-1 having a distance value of zero. If noise 47d occurs on ground 45d, filtering results in the pixels in noise 47d becoming corrected pixels 49d-2 having the same distance value as the pixels in ground 45d. If noise 47d occurs on object 43, filtering results in the pixels in noise 47d becoming corrected pixels 49d-3 having the same distance value as the pixels in object 43. However, if multiple noise elements 47d are close to or densely packed together, the pixels in noise 47d are unlikely to be corrected and may remain as they are, even after filtering.

[0055] FIG. 13 is a diagram showing a pixel group 40e' after the pixel group 40e in FIG. 8 has been processed by the noise filter 25. The distance values ​​of most of the pixels in the noise 47e in FIG. 8 are corrected by filtering according to the pixel values ​​of their surrounding pixels, and the pixels in the noise 47e become corrected pixels having corrected distance values. In the example of FIG. 13, if the pixels surrounding the noise 47e have distance values ​​of zero, the pixels in the noise 47e become corrected pixels 49e having distance values ​​of zero by filtering. However, if multiple noises 47e are close to or densely packed together, the pixels in the noise 47e are unlikely to be corrected even after filtering, and may remain as they are.

[0056] Figure 14 shows a distance image 40' obtained by combining pixel groups 40a' to 40e' of Figures 9 to 13. Distance image 40' includes noise 47 corresponding to noise 47a to 47e of Figures 9 to 13, defective pixel 48 corresponding to defective pixels 48a to 48e of Figures 9 to 13, and corrected pixel 49 corresponding to corrected pixels 49a to 49e of Figures 9 to 13.

[0057] If the same pixel in different pixel groups is corrected to have a non-zero distance value by filtering (or by interpolation, as described below), the distance values ​​of the pixels in these pixel groups may conflict with each other during compositing. In this case, the reliability of the pixel is calculated based on the number of neighboring pixels, and the most likely distance value is selected.

[0058] As described with reference to Figures 9 to 13, by processing pixel groups 40a to 40e individually, the density of noise 47 is reduced, making it easier to reduce or remove noise 47. Furthermore, because the density of noise 47 is reduced, even when filtering is performed, it is less likely to interfere with objects 41 to 44. This makes it possible to reliably reduce noise in the distance image without amplifying the noise or enlarging defective pixels.

[0059] 9 to 13, the filter parameters of each of pixel groups 40a' to 40e' are set so that the performance of noise filter 25 decreases as the representative distance value of the pixel group increases. Objects at a long distance from image capture device 1 are more susceptible to the influence of defective pixels 48 than objects at a close distance. As described above, by setting the performance of noise filter 25 according to distance, it is possible to make it difficult for defective pixels 48 to expand in objects at a long distance from image capture device 1.

[0060] [Advantages of the first embodiment] An image processing device 2 according to one embodiment of the present disclosure includes an input interface 21, an image divider 24, a noise filter 25, and an image synthesizer 26. The input interface 21 acquires a first distance image including a plurality of pixels each indicating a distance value from the imaging device 1 to each point on an object. The image divider 24 divides the first distance image into a plurality of pixel groups based on the distance values ​​of each pixel, such that each pixel group includes a pixel having a distance value included in one of a plurality of distance intervals that are different from each other. The noise filter 25 processes the plurality of pixel groups individually using a plurality of filter parameters that are different from each other for each pixel group, thereby reducing noise in the plurality of pixel groups. The image synthesizer 26 synthesizes the plurality of pixel groups processed by the noise filter 25 to generate a second distance image.

[0061] This makes it possible to reliably reduce noise in the distance image without amplifying noise or enlarging defective pixels.

[0062] According to an embodiment of the present disclosure, the image processing device 2 includes a plurality of pixel groups each having a representative distance value that represents the distance values ​​of the pixels included in the pixel group. In this case, the plurality of filter parameters may be set so that the noise reduction performance of the noise filter 25 decreases as the representative distance value of the pixel group increases.

[0063] This makes it possible to make it difficult for defective pixels in an object that is far away from the imaging device 1 to become enlarged.

[0064] The image processing device 2 according to one embodiment of the present disclosure may include at least one processor that operates as an image divider 24, a noise filter 25, and an image combiner .

[0065] This allows the image processing device 2 to be implemented using one or more general-purpose or special-purpose processors.

[0066] An image processing method according to one embodiment of the present disclosure includes acquiring a first distance image including a plurality of pixels each indicating a distance value from an imaging device 1 to each point on an object. The image processing method includes dividing the first distance image into a plurality of pixel groups based on the distance value of each pixel, such that each of the plurality of pixel groups includes pixels having a distance value included in one of a plurality of distance intervals that are different from each other. The image processing method also includes reducing noise in the plurality of pixel groups by individually processing the plurality of pixel groups using a plurality of filter parameters that are different from each other for each pixel group. The image processing method also includes generating a second distance image by combining the plurality of pixel groups processed in the noise reduction step.

[0067] This makes it possible to reliably reduce noise in the distance image without amplifying noise or enlarging defective pixels.

[0068] [Second embodiment] [Configuration of the second embodiment] 15 is a schematic diagram showing the configuration of an image processing device 2A according to a second embodiment. The image processing device 2A includes a processing circuit 20A instead of the processing circuit 20 in FIG. 1, and further includes a storage device 33. The processing circuit 20A includes an image divider 24A and an image combiner 26A instead of the image divider 24 and the image combiner 26 in FIG. 1, and further includes an interpolator 31 and an interpolation controller 32.

[0069] 1, image divider 24A acquires a distance image from imaging device 1 via input interface 21 and divides the distance image into multiple pixel groups based on the distance value of each pixel. Image divider 24A sends some of the divided pixel groups to interpolator 31 and sends the remaining pixel groups to noise filter 25. Image divider 24A may, for example, send pixel groups having relatively small representative distance values ​​to noise filter 25 and send pixel groups having relatively large representative distance values ​​to interpolator 31.

[0070] The interpolator 31 processes at least one of the plurality of pixel groups using predetermined interpolation parameters to interpolate missing pixels of the pixel group. When processing multiple pixel groups, the interpolator 31 may process the multiple pixel groups individually using multiple different interpolation parameters for each pixel group to reduce noise in the multiple pixel groups. In this case, the interpolator 31 may process the multiple pixel groups sequentially or in parallel.

[0071] The storage device 33 stores in advance a plurality of interpolation parameters for one or more pixel groups. The plurality of interpolation parameters are set, for example, so that the performance of the interpolator 31 in interpolating missing pixels increases as the representative distance value of the pixel group increases.

[0072] The interpolation controller 32 reads out the interpolation parameters from the storage device 33 and sets them in the interpolator 31. When processing a plurality of pixel groups, the interpolation controller 32 sets the interpolation parameters in the interpolator 31 for each pixel group.

[0073] The image synthesizer 26A synthesizes the pixel group processed by the noise filter 25 and the pixel group processed by the interpolator 31 to generate a distance image.

[0074] The processing circuit 20A may include multiple dedicated circuits corresponding to the image divider 24A, the noise filter 25, the image combiner 26A, the filter controller 27, the interpolator 31, and the interpolation controller 32, respectively. Alternatively, the processing circuit 20A may include one or more general-purpose or dedicated processors (e.g., digital signal processors) that operate as the image divider 24A, the noise filter 25, the image combiner 26A, the filter controller 27, the interpolator 31, and the interpolation controller 32 by executing predetermined programs. The storage devices 23 and 33 may be provided separately or may be integrated with each other.

[0075] [Operation of the second embodiment] Fig. 16 is a flowchart showing noise reduction processing executed by processing circuit 20A in Fig. 15. The noise reduction processing in Fig. 16 includes steps S3A, S6A, and S7A instead of steps S3, S6, and S7 in Fig. 2, and further includes steps S11 to S15.

[0076] The processing circuit 20A executes steps S1 to S2 in FIG. 16 in the same manner as steps S1 to S2 in FIG.

[0077] Next, processing circuit 20A selects one pixel group from the pixel groups to be subjected to noise reduction (step S3A). The pixel group to be subjected to noise reduction may be, for example, pixel groups 40a to 40c having relatively small representative distance values ​​among pixel groups 40a to 40e in FIGS.

[0078] Next, the processing circuit 20A executes steps S4 to S5 in FIG. 16 in the same manner as steps S4 to S5 in FIG.

[0079] Next, processing circuit 20A determines whether or not processing has been performed to reduce noise for all pixel groups that are the target of noise reduction (step S6A), and if YES, proceeds to step S11, and if NO, proceeds to step S7A.

[0080] Next, processing circuit 20A selects the next unselected pixel group from the pixel groups that are targets of noise reduction (step S7A), and then repeats steps S4, S5, and S6A.

[0081] Steps S3A, S5, S6A, and S7A show the operation of the processing circuit 20A as the noise filter 25, and step S4 shows the operation of the processing circuit 20A as the filter controller 27.

[0082] Next, processing circuit 20A selects one pixel group from the pixel groups to be interpolated (step S11). The pixel group to be interpolated may be, for example, pixel groups 40d to 40e having relatively large representative distance values ​​from pixel groups 40a to 40e in FIGS. 4 to 8.

[0083] Next, processing circuit 20A selects and sets interpolation parameters corresponding to the pixel group selected in step S11 (step S12).

[0084] Processing circuitry 20A then processes the pixel group so as to interpolate missing pixels in the pixel group using the set interpolation parameters (step S13).

[0085] Next, processing circuit 20A determines whether or not processing has been performed to interpolate missing pixels for all pixel groups to be interpolated (step S14), and if YES, proceeds to step S8, and if NO, proceeds to step S15.

[0086] Next, processing circuit 20A selects the next unselected pixel group from the pixel groups to be interpolated (step S15), and then repeats steps S12 to S14.

[0087] Steps S11 and S13 to S15 show the operation of the processing circuit 20A as the interpolator 31, and step S12 shows the operation of the processing circuit 20A as the interpolation controller 32.

[0088] Next, the processing circuit 20A executes steps S8 to S9 in FIG. 16 in the same manner as steps S8 to S9 in FIG.

[0089] 16 shows a case where steps S11 to S15 are executed after steps S3A to S7, but processing circuit 20A may execute steps S11 to S15 before steps S3A to S7. Furthermore, processing circuit 20A may execute steps S11 to S15 in parallel with steps S3A to S7.

[0090] According to the noise reduction process of FIG. 16, it is possible to reduce noise in the distance image and to interpolate missing pixels in the distance image.

[0091] By processing multiple pixel groups individually, the density of defective pixels is reduced, making it easier to interpolate the defective pixels. Furthermore, because the density of defective pixels is reduced, even if interpolation is performed, they are less likely to interfere with the object. This allows for reliable interpolation of defective pixels in the range image without amplifying noise or enlarging the defective pixels.

[0092] An object at a long distance from the imaging device 1 is less susceptible to noise, but an object at a short distance from the imaging device 1 is more susceptible to noise. On the other hand, an object at a long distance from the imaging device 1 is more susceptible to missing pixels, but an object at a short distance from the imaging device 1 is less susceptible to missing pixels. Therefore, by selectively applying filtering or interpolation according to the distance from the imaging device 1, it is possible to effectively reduce noise in an object at a short distance from the imaging device 1, and to effectively interpolate missing pixels in an object at a long distance from the imaging device 1, for example.

[0093] Furthermore, by setting the interpolation parameters so that the performance of the interpolator 31 increases as the representative distance value of the pixel group increases, it is possible to effectively interpolate missing pixels in an object that is at a long distance from the imaging device 1.

[0094] [Advantages of the second embodiment] The image processing device 2A according to one embodiment of the present disclosure may further include an interpolator 31 that processes at least one of the plurality of pixel groups using predetermined interpolation parameters to interpolate missing pixels in the pixel group. In this case, the image synthesizer 26A synthesizes the pixel group processed by the noise filter 25 and the pixel group processed by the interpolator 31 to generate a second distance image.

[0095] This makes it possible to reduce noise in the distance image and to interpolate missing pixels in the distance image.

[0096] According to an embodiment of the image processing device 2A of the present disclosure, each of a plurality of pixel groups has a representative distance value that represents the distance values ​​of the pixels included in that pixel group. In this case, the interpolation parameters may be set so that the performance of the interpolator 31 in interpolating missing pixels increases as the representative distance value of the pixel group increases.

[0097] This allows for effective interpolation of missing pixels in an object that is far away from the image capturing device 1.

[0098] [Third embodiment] [Configuration of the third embodiment] Fig. 17 is a schematic diagram showing the configuration of an image processing device 2B according to the third embodiment. The image processing device 2B includes a processing circuit 20B instead of the processing circuit 20 in Fig. 1. The processing circuit 20B includes an image recognizer 34 in addition to the components of the processing circuit 20 in Fig. 1.

[0099] The image recognizer 34 recognizes a predetermined object, such as a person or a vehicle, in each of the multiple pixel groups processed by the noise filter 25. By performing image recognition on pixel groups corresponding to a predetermined distance interval, it is possible to recognize objects included in the distance image with higher accuracy than when image recognition is performed on a distance image synthesized by the image synthesizer 26.

[0100] The processing circuit 20B may include multiple dedicated circuits corresponding respectively to the image divider 24, the noise filter 25, the image synthesizer 26, the filter controller 27, and the image recognizer 34. Alternatively, the processing circuit 20B may include one or more general-purpose or dedicated processors (e.g., digital signal processors) that operate as the image divider 24, the noise filter 25, the image synthesizer 26, the filter controller 27, and the image recognizer 34 by executing predetermined programs.

[0101] [Advantages of the third embodiment] The image processing device 2B according to the embodiment of the present disclosure may further include a first image recognizer 34 that recognizes a predetermined object in each of the plurality of pixel groups processed by the noise filter 25.

[0102] This allows objects included in the distance image to be recognized with higher accuracy than when image recognition is performed on the distance image.

[0103] [Fourth embodiment] [Configuration of the fourth embodiment] Fig. 18 is a schematic diagram showing the configuration of an image processing device 2C according to the fourth embodiment. The image processing device 2C includes a processing circuit 20C instead of the processing circuit 20 in Fig. 1. The processing circuit 20C includes a filter controller 27C instead of the filter controller 27 in Fig. 1, and further includes an image recognizer 35.

[0104] The image recognizer 35 recognizes a predetermined object in each of the plurality of pixel groups before they are processed by the noise filter 25 .

[0105] The filter controller 27C sets filter parameters to be applied to a pixel group including an object based on the distance from the image capture device 1 to the object recognized by the image recognizer 35. For example, the filter controller 27C may set the filter parameters according to the distance of the recognized object so as to reduce the performance of the noise filter 25 for a pixel group including the object and increase the performance of the noise filter 25 for a pixel group not including the object. This makes it possible to adjust the performance of the noise filter 25 so as not to blur important objects.

[0106] The filter controller 27C may also set the filter parameters depending on, for example, the apparent size of a recognized object instead of its distance.

[0107] By adaptively setting the filter parameters according to the distance from the image capture device 1 to the object, noise in the distance image can be reduced more reliably.

[0108] The processing circuit 20C may include multiple dedicated circuits corresponding respectively to the image divider 24, the noise filter 25, the image synthesizer 26, the filter controller 27C, and the image recognizer 35. Alternatively, the processing circuit 20C may include one or more general-purpose or dedicated processors (e.g., digital signal processors) that operate as the image divider 24, the noise filter 25, the image synthesizer 26, the filter controller 27C, and the image recognizer 35 by executing predetermined programs.

[0109] [Effects of the fourth embodiment] The image processing device 2C according to one embodiment of the present disclosure may further include a second image recognizer 35 and a filter controller 27C. In this case, the second image recognizer 35 recognizes a predetermined object in each of the plurality of pixel groups before being processed by the noise filter 25. Furthermore, the filter controller 27C sets filter parameters to be applied to the pixel groups including the object recognized by the second image recognizer 35, based on the distance from the imaging device 1 to the object.

[0110] This makes it possible to more reliably reduce noise in the distance image.

[0111] [Other embodiments] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which appropriate modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments.

[0112] Therefore, other embodiments will be exemplified below.

[0113] The image processing device 2 or the like may process the distance image acquired from the imaging device 1 in real time, or may read and process the distance image that has been temporarily stored in an external storage device.

[0114] When the image processing device 2 or the like includes a plurality of filter circuits connected in parallel to each other and each having predetermined filter parameters set thereto, the storage device 23 and the filter controller 27 may be omitted.

[0115] The filter parameters may be set, for example, so that the performance of the noise filter 25 gradually increases as the representative distance value of the pixel group increases, and then gradually decreases.

[0116] The filter parameters may also be set to take into account the lens characteristics of the imaging device 1.

[0117] The distance interval may be divided into a predetermined number of intervals, or may be divided adaptively based on the distance distribution of the object.

[0118] In the example of the second embodiment, a case has been described in which only one of noise filtering and interpolation is performed on each pixel group, but both noise filtering and interpolation may be performed on at least one pixel group.

[0119] The image processing device 2 and the like may also be applied to a system that performs three-dimensional measurements to digitize information for on-site analysis or optimization. For example, when modeling the volume of cargo using distance images to digitize the loading rate of cargo on carts or truck beds in a logistics warehouse, noise in the acquired distance images can reduce the accuracy of the three-dimensional measurements, making accurate modeling impossible. On the other hand, an image processing device and an image processing method according to one embodiment of the present disclosure can reliably reduce noise in distance images, thereby achieving accurate three-dimensional measurements.

[0120] The embodiments described above may be combined with each other. For example, the image processing device 2C of Fig. 18 may further include the image recognizer 34 of Fig. 17. Furthermore, the image processing device 2A of Fig. 15 may include the image recognizer 34 of Fig. 17 and at least one of the filter controller 27C and the image recognizer 35 of Fig. 18.

[0121] As described above, the embodiments have been described as examples of the technology in the present disclosure, and for that purpose, the accompanying drawings and detailed description have been provided.

[0122] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0123] Furthermore, since the above-described embodiments are intended to illustrate the technology of the present disclosure, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0124] An image processing device and an image processing method according to an aspect of the present disclosure are applicable to reducing random noise in a range image. [Explanation of symbols]

[0125] 1. Imaging device 2, 2A to 2C Image processing device 20, 20A~20C Processing circuit 21 Input interface (I / F) 22 Output interface (I / F) 23 Storage device 24,24A Image divider 25 Noise Filter 26,26A Image synthesizer 27,27C Filter Controller 31 Interpolator 32 Interpolation Controller 33 Storage device 34,35 Image Recognizer

Claims

1. an input interface for acquiring a first distance image including a plurality of pixels each indicating a distance value from the imaging device to each point on the object; an image divider that divides the first distance image into a plurality of pixel groups based on the distance values ​​of each pixel, such that each pixel group includes a pixel having a distance value that is included in one of a plurality of distance intervals that are different from each other; a noise filter that processes the plurality of pixel groups individually using a plurality of filter parameters that are different from each other for each of the plurality of pixel groups to reduce noise in the plurality of pixel groups; a second image recognizer that recognizes a predetermined object in each of the plurality of pixel groups before being processed by the noise filter; a filter controller that sets filter parameters to be applied to a pixel group including an object recognized by the second image recognizer based on a distance from the imaging device to the object; an image combiner that combines the plurality of pixel groups processed by the noise filter to generate a second distance image, Image processing device.

2. each of the plurality of pixel groups has a representative distance value that represents the distance values ​​of pixels included in the pixel group; the plurality of filter parameters are set so that the performance of the noise filter in reducing noise decreases as the representative distance value of the pixel group increases; 2. The image processing device according to claim 1.

3. at least one processor operating as the image divider, the noise filter, and the image combiner; 3. The image processing device according to claim 1.

4. the image processing device further comprises an interpolator that processes at least one of the plurality of pixel groups using predetermined interpolation parameters to interpolate missing pixels of the pixel group; the image combiner combines the pixel group processed by the noise filter and the pixel group processed by the interpolator to generate the second distance image.

4. The image processing device according to claim 1.

5. each of the plurality of pixel groups has a representative distance value that represents the distance values ​​of pixels included in the pixel group; the interpolation parameters are set such that the performance of the interpolator in interpolating missing pixels increases as the representative distance value of the pixel group increases; 5. The image processing device according to claim 4.

6. a first image recognizer configured to recognize a predetermined object in each of the plurality of pixel groups processed by the noise filter; 6. An image processing device according to claim 1.

7. acquiring a first distance image including a plurality of pixels each indicating a distance value from an imaging device to each point on an object; dividing the first distance image into a plurality of pixel groups based on the distance values ​​of each pixel, such that each pixel group includes pixels having distance values ​​included in one of a plurality of distance intervals that are different from each other; reducing noise in the plurality of pixel groups by individually processing the plurality of pixel groups using a plurality of different filter parameters for each of the plurality of pixel groups; recognizing a predetermined object in each of the plurality of pixel groups before being processed in the noise reducing step; setting a filter parameter to be applied to a pixel group including the recognized object based on a distance from the imaging device to the recognized object; and generating a second distance image by combining the plurality of pixel groups processed in the noise reduction step. Image processing methods.

Images