Image processing apparatus and image processing method

By deriveing ​​and generating static pixel values ​​in the image processing device, the problem of unusing static pixels in the stereoscopic measurement device is solved, and the generation of static images and image quality improvement is achieved.

CN119013698AInactive Publication Date: 2024-11-22PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380031925.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-31
Filing Date
2023-03-29
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art has failed to effectively utilize the pixel values ​​of static pixels other than the moving region in the stereoscopic measurement device, and does not provide a method of generating a static image.

Method used

An image processing device and method are designed, and the control unit processes images captured at a predetermined time, derives static pixel values ​​based on the change of pixel values ​​over time, and generates a static image representing the static pixel values ​​of each pixel.

Benefits of technology

It realizes the generation of static images using the pixel values ​​of static pixels, which reduces noise components, improves image quality, and reduces the load of stereo matching processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013698A_ABST
    Figure CN119013698A_ABST
Patent Text Reader

Abstract

A static pixel value deriving unit (100) derives a static pixel value (SS) of a static pixel included in an image (P0) on the basis of a change over time in the pixel value (S) of each pixel in the image (P0). A still image generation unit (103) generates a still image (P10) indicating the still pixel value (SS) for each pixel on the basis of the still pixel values (SS).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The technology disclosed here relates to an image processing device and an image processing method. Background Art

[0002] Patent document 1 discloses a stereo measurement device. The stereo measurement device includes a first motion region extraction unit, a second motion region extraction unit, and a stereo matching unit. The first motion region extraction unit extracts a motion region from a first image captured by a first camera. The second motion region extraction unit extracts a motion region from a second image captured by a second camera. The stereo matching unit obtains distance information related to an object shown by the motion region by performing stereo matching only on the motion region extracted from the first image and the second image.

[0003] Patent Document 1: Japanese Patent Publication No. 2009-68935 Summary of the invention

[0004] -Technical problem to be solved by the invention-

[0005] Patent Document 1 does not disclose or suggest how to utilize pixels (static pixels) in the remaining areas of the first and second images other than the motion area, nor does it disclose or suggest how to generate a static image using the pixel values ​​of the static pixels.

[0006] -Technical solutions for solving technical problems-

[0007] The technology disclosed here is an image processing device that processes images captured by a camera at predetermined intervals. The image processing device includes a control unit connected to the camera, the control unit includes a static pixel value extraction unit and a static image generation unit. The static pixel value extraction unit extracts the static pixel values ​​of the static pixels contained in the image based on the change of the pixel value of each pixel of the image over time, and the static image generation unit generates a static image representing the static pixel values ​​of each pixel based on the static pixel values.

[0008] The technology disclosed here is an image processing method, which processes images captured by a camera at specified intervals. The image processing method includes: a step of deriving static pixel values ​​of static pixels contained in the image based on changes in the pixel value of each pixel of the image over time; and a step of generating a static image representing the static pixel value of each pixel based on the static pixel value.

[0009] -Effects of the Invention-

[0010] According to the technology disclosed herein, a static image can be generated using the pixel values ​​of static pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of the structure of an image processing apparatus according to an exemplary embodiment;

[0012] Figure 2 is a block diagram of the functional structure of an exemplary control unit;

[0013] Figure 3 is a flow chart for explaining the operation of the image processing device;

[0014] Figure 4 is a flow chart for explaining the operation of the image processing device;

[0015] Figure 5 This is a conceptual diagram of an example image processing flow. DETAILED DESCRIPTION

[0016] The following is a detailed description of the embodiments with reference to the accompanying drawings. It should be noted that the same or corresponding parts are indicated by the same symbols in the drawings, and the description thereof will not be repeated.

[0017] (Image processing device)

[0018] Figure 1 The configuration of the image processing device 10 according to the exemplary embodiment is shown below. The image processing device 10 processes images acquired at predetermined time intervals. The image processing device 10 includes a control unit 11 and a storage unit 12 .

[0019] In this example, a first camera 21 and a second camera 22 are connected to the image processing device 10. For example, the first camera 21 and the second camera 22 are fixed at predetermined locations in a factory and face predetermined directions to perform image capturing.

[0020] The first camera 21 and the second camera 22 are stereo cameras that simultaneously photograph the photographed area from different viewpoints. The first camera 21 acquires a first image P1 by photographing the photographed area at predetermined intervals. The second camera 22 acquires a second image P2 by photographing the photographed area at predetermined intervals. For example, the first camera 21 and the second camera 22 are composed of general cameras having image sensors such as CCD (Charge Coupled Device) image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors. The first camera 21 and the second camera 22 are an example of a photographing device that photographs the photographed area at predetermined intervals.

[0021] The image processing device 10 inputs the first image P1 acquired by the first camera 21 and the second image P2 acquired by the second camera 22. The Nth (N is an integer) second image P2 on the time axis and the Nth first image P1 on the time axis are images obtained at the same time. Hereinafter, the first image P1 and the second image P2 are collectively referred to as "image P0". The pixel value of image P0 is a brightness value (signal amount) indicating the brightness level of the photographed area.

[0022] 〔Control Department〕

[0023] The control unit 11 is connected to the components of the image processing device 10 through a signal line, and controls the components of the image processing device 10. In this example, the control unit 11 is connected to the storage unit 12 through a signal line. In addition, the control unit 11 is connected to the first camera 21 and the second camera 22 through a signal line, and controls the first camera 21 and the second camera 22. The control unit 11 stores the first image P1 sent from the first camera 21 and the second image P2 sent from the second camera 22 in the storage unit 12.

[0024] For example, the control unit 11 includes a processor and a memory storing a program for operating the processor. The various functions of the control unit 11 are realized by executing the program by the processor. In other words, the control unit 11 has various functional blocks for realizing various functions. It should be noted that the control unit 11 is an example of a computer.

[0025] 〔Storage Department〕

[0026] The storage unit 12 stores various data and information. For example, the storage unit 12 stores the first image P1 sent from the first camera 21, the second image P2 sent from the second camera 22, the static image P10, the static pixel value SS, etc. It should be noted that the image P0 (specifically, the first image P1 and the second image P2) is associated with the time when the image P0 is acquired. The static image P10 and the static pixel value SS will be described in detail later.

[0027] 〔Processing by the control unit〕

[0028] The control unit 11 performs various processes. In this example, the control unit 11 performs static pixel value derivation processing, static image generation processing, stereo matching processing, feature image generation processing, filling processing, recognition processing, and three-dimensional data generation processing. The static pixel value derivation processing includes a judgment processing and a derivation processing. It should be noted that the judgment processing is performed based on the first image P1 and the second image P2, respectively. The derivation processing and the static image generation processing are performed based on at least one of the first image P1 and the second image P2. The processing performed by the control unit 11 is an example of an image processing method.

[0029] like Figure 2 As shown, in this example, the control unit 11 includes a static pixel value derivation unit 100 , a static image generation unit 103 , a stereo matching unit 104 , a feature image generation unit 105 , a filling unit 106 , a recognition unit 107 , and a three-dimensional data generation unit 108 .

[0030] The static pixel value derivation unit 100 performs a static pixel value derivation process. The static pixel value derivation unit 100 includes a judgment unit 101 that performs a judgment process and an derivation unit 102 that performs a derivation process. The static image generation unit 103 performs a static image generation process. The stereo matching unit 104 performs a stereo matching process. The feature image generation unit 105 performs a feature image generation process. The filling unit 106 performs a filling process. The recognition unit 107 performs a recognition process. The three-dimensional data generation unit 108 performs a three-dimensional data generation process. It should be noted that the judgment unit 101 performs processing based on the first image P1 and the second image P2, respectively. The derivation unit 102 and the static image generation unit 103 perform processing based on at least one of the first image P1 and the second image P2.

[0031] In the static pixel value derivation process, the control unit 11 (static pixel value derivation unit 100) derives the static pixel value SS of the static pixel included in the image P0 based on the change of the pixel value S of each pixel of the image P0 over time. Specifically, in the static pixel value derivation process, a determination process and a derivation process are performed.

[0032] In the determination process, the control unit 11 (determination unit 101 ) determines, for each pixel of the image P0 , whether the pixel is a dynamic pixel or a static pixel based on a change in the pixel value S of the pixel over time.

[0033] During the export process, the control unit 11 (export unit 102) exports the static pixel value SS of each pixel of the image P0 based on the pixel value S of the pixel at the past moment and the pixel value S of the pixel at the current moment when the pixel is judged to be a static pixel through the judgment process.

[0034] In the still image generation process, the control unit 11 (still image generation unit 103) generates a still image P10 representing the still pixel value SS of each pixel (the still pixel value SS derived by the derivation process) based on the still pixel value SS derived for each pixel of the image P0 by the derivation process.

[0035] [Operation of the image processing device (determination processing, derivation processing, and still image generation processing)]

[0036] Next, refer to Figure 3 , the operation of the image processing device 10 (determination processing, derivation processing, and static image generation processing) is described. Figure 3 In the above, step S11, step S12 and step S13 constitute a determination step S1 corresponding to the determination process. Steps S14, S16 and S17 constitute a derivation step S2 corresponding to the derivation process. Step S18 constitutes a still image generation step S3 corresponding to the still image generation process. For example, each time the image P0 is acquired, the control unit 11 performs the following process.

[0037] <Step S11>

[0038] The control unit 11 selects a pixel to be processed from the image P0. Then, the control unit 11 acquires the pixel value S of the selected pixel. Hereinafter, the pixel selected as the processing target in step S11 is referred to as a "target pixel".

[0039] <Step S12>

[0040] Next, the control unit 11 acquires the static pixel value SS of the target pixel. In this example, the control unit 11 reads the static pixel value SS of the target pixel stored in the storage unit 12.

[0041] <Step S13>

[0042] Next, the control unit 11 (determination unit 101) determines whether the target pixel is a dynamic pixel or a static pixel based on the temporal change in the pixel value S of the target pixel.

[0043] Specifically, when the amount of change of the pixel value S of the object pixel over time exceeds a preset threshold, the control unit 11 determines that the object pixel is a dynamic pixel. On the other hand, when the amount of change of the pixel value S of the object pixel over time does not exceed the threshold, the control unit 11 determines that the object pixel is a static pixel. It should be noted that in this example, the control unit 11 derives the absolute value of the difference between the static pixel value SS of the object pixel and the pixel value S of the object pixel as the "amount of change of the pixel value S of the object pixel over time".

[0044] In addition, the above threshold is set individually for each pixel of the image P0. In this example, the control unit 11 (determination unit 101) sets the threshold set for each pixel of the image P0 to a value corresponding to the static pixel value SS of the pixel. Specifically, the control unit 11 sets the square root of the static pixel value SS as the threshold.

[0045] When the target pixel is not a dynamic pixel (when the target pixel is a static pixel), the process of step S14 is performed. On the other hand, when the target pixel is a dynamic pixel, the process of step S17 is performed.

[0046] <Step S14>

[0047] When it is determined in step S13 that the object pixel is a static pixel, the control unit 11 (derivation unit 102) derives the static pixel value SS of the object pixel based on the "static pixel value SS of the object pixel stored in the storage unit 12" and the "pixel value S of the object pixel", and updates the static pixel value SS of the object pixel stored in the storage unit 12 to the derived static pixel value SS.

[0048] In this example, the control unit 11 sets the average value obtained by averaging the “static pixel value SS of the target pixel stored in the storage unit 12 ” and the “pixel value S of the target pixel” as the new static pixel value SS of the target pixel.

[0049] Here, if the static pixel value SS derived from the object pixel of the Nth image P0 on the time axis is set to “SS (N) ”, and the static pixel value SS of the object pixel stored in the storage unit 12 (the static pixel value derived based on the object pixel of the N-1th image P0 on the time axis) is set to “SS (n-1) ", the new static pixel value SS (N) It is expressed as the following formula 1.

[0050] [Mathematical formula 1]

[0051]

[0052] In addition, if the "image P0 in which the object pixel is determined to be a static pixel" immediately following the "image P0 in which the object pixel is determined to be a dynamic pixel" on the time axis (the image P0 in which the determination result of the object pixel changes from a dynamic pixel to a static pixel) is set as the "first image P0", and the pixel values ​​S of the object pixels of the first to N-1th images P0 are set as "S (1) ~S (n-1) ", then if the object pixel is determined to be a static pixel from the first image P0 to the N-1th image P0, the static pixel value SS derived based on the object pixel of the N-1th image P0 is (n-1) It is expressed as the following formula 2.

[0053] [Mathematical formula 2]

[0054]

[0055] Thus, the N-1th static pixel value SS (n-1) is N-1 pixel value S (1) ~S (n-1) In other words, to derive the N-1th static pixel value SS (n-1) , using N-1 pixel values ​​S (1) ~S(n-1) .

[0056] <Step S15>

[0057] Next, the control unit 11 determines whether the processing of all pixels included in the image P0 has been completed. If the processing of all pixels has been completed, the processing of step S18 is performed. On the other hand, if the processing of all pixels has not been completed, the processing of step S11 is performed to select the pixel to be processed next.

[0058] <Step S16>

[0059] When it is determined in step S13 that the target pixel is a dynamic pixel, the control unit 11 restores the static pixel value SS of the target pixel to an initial value. In this example, the initial value is zero.

[0060] <Step S17>

[0061] When it is determined in step S13 that the object pixel belongs to a dynamic pixel, the control unit 11 (determination unit 101) stores the address of the object pixel determined to belong to the dynamic pixel in association with the image P0 in the storage unit 12. In other words, the control unit 11 stores the dynamic pixel information in the storage unit 12, and the dynamic pixel information indicates which pixel of which image P0 belongs to the dynamic pixel. Then, the process of step S15 is performed.

[0062] <Step S18>

[0063] The control unit 11 (still image generation unit 103 ) generates a still image P10 representing the still pixel value SS of each pixel based on the still pixel value SS derived for each pixel of the image in the above steps S11 to S17 .

[0064] The arrangement of pixels in the static image P10 is the same as the arrangement of pixels in the image P0. For example, in the case where the image P0 includes pixels in m rows and n columns (m and n are integers), the static image P10 also includes pixels in m rows and n columns. Then, in the case where the object pixel is a pixel in the i-th row (i is an integer from 1 to m) and the j-th column (j is an integer from 1 to n), the control unit 11 sets the static pixel value SS of the i-th row and the j-th row of the static image P10 to the newly derived static pixel value SS of the object pixel (or the static pixel value SS restored to the initial value).

[0065] It should be noted that the static image P10 is associated with the image P0 and stored in the storage unit 12. For example, the Nth static image P10 on the time axis is associated with the Nth image P0 on the time axis. The Nth static image P10 associated with the Nth image P0 is a collection of static pixel values ​​SS derived from the pixels of the Nth image P0.

[0066] The static image P10 contains not only static elements but also dynamic elements with less movement. Examples of static elements include backgrounds and stationary objects. Examples of dynamic elements include moving objects. For example, in a factory, walls and floors in the factory belong to the background, housings of equipment installed in the factory belong to stationary objects, and workers, robot arms, and objects in the process of transportation belong to moving objects.

[0067] [Operation status of image processing device (stereo matching processing, etc.)]

[0068] Next, refer to Figure 4 The operation of the image processing device 10 (stereo matching processing, etc.) will be described below. Figure 3 When the shown processing is completed, the control unit 11 performs the following processing.

[0069] It should be noted that the following references are appropriate Figure 5 For explanation. Figure 5 In the example, the first image P1 and the second image P2 represent a photographed area, in which a worker stands in front of a device installed in a factory. In this example, the dynamic element D1 is a worker.

[0070] <Step S4: Stereo Matching Step>

[0071] The control unit 11 (stereo matching unit 104) performs stereo matching processing. In the stereo matching processing, the control unit 11 performs stereo matching on pixels of the first image P1 that are determined to be dynamic pixels in the determination processing and pixels of the second image P2 that are determined to be dynamic pixels in the determination processing, thereby acquiring disparity information related to the dynamic element D1 shown in the first image P1 and the second image P2.

[0072] It should be noted that the control unit 11 detects “pixels determined to be dynamic pixels” from the first image P1 and the second image P2 based on the dynamic pixel information (information indicating which pixel of which image belongs to a dynamic pixel) stored in the storage unit 12 .

[0073] The stereo matching process can adopt a known method. For example, the control unit 11 detects block areas with a high degree of consistency with the block areas selected in the first image P1 from the second image P2 and obtains the parallax of these block areas. By repeatedly performing this process, parallax information is obtained.

[0074] <Step S5: Feature Image Generating Step>

[0075] Next, the control unit 11 (characteristic image generation unit 105) performs a characteristic image generation process. In the characteristic image generation process, the control unit 11 generates a characteristic image P3 (see FIG. 105 ) representing the contour portion D2 of the dynamic element D1 based on the parallax information obtained by the stereo matching process. Figure 5 ).

[0076] The feature image generation process can adopt a known method. For example, the control unit 11 generates a distance image representing the distance of each pixel (the distance in the depth direction of the image) as the feature image P3 based on the parallax information obtained by the stereo matching process. In the distance image, the contour part D2 of the dynamic element D1 is displayed as an edge line, but since the distance values ​​are approximately the same, the inside of the contour part D2 is difficult to appear in the image.

[0077] It should be noted that the Nth feature image P3 on the time axis is generated based on disparity information, which is information obtained by stereo matching of "dynamic pixels of the Nth first image P1 on the time axis" and "dynamic pixels of the Nth second image P2 on the time axis".

[0078] <Step S6: Filling Step>

[0079] Next, the control unit 11 (filling unit 106) performs a filling process. In the filling process, the control unit 11 fills the image portion PP inside the outline portion D2 of the dynamic element D1 shown in the characteristic image P3 based on the static image P10 generated by the static image generation process. The characteristic image P3 is generated by the characteristic image generation process. In this way, a filled image P4 (see Figure 5 ).

[0080] The filling process can be performed by a known method. For example, the control unit 11 extracts an image portion PP having a contour portion that is highly consistent with the contour portion D2 (edge ​​line) of the dynamic element D1 shown in the feature image P3 from the static image P10, and embeds the extracted image portion PP into the contour portion D2 of the dynamic element D1 shown in the feature image P3.

[0081] <Step S7: Identification Step>

[0082] Next, the control unit 11 (recognition unit 107) performs recognition processing. In the recognition processing, the control unit 11 recognizes the dynamic element D1 from the feature image (filled image P4) processed by the fill processing.

[0083] The recognition process can adopt a known method. For example, the control unit 11 recognizes the dynamic element D1 from the fill image P4 using a model obtained by machine learning such as deep learning.

[0084] <Step S8: Three-dimensional data generation step>

[0085] Next, the control unit 11 (3D data generation unit 108 ) performs 3D data generation processing. In the 3D data generation processing, the control unit 11 generates 3D data based on the first image P1 or the second image P2 , the parallax information obtained by the stereo matching processing, and the recognition result of the recognition processing.

[0086] The three-dimensional data generation process can adopt a known method. For example, the control unit 11 generates three-dimensional data by adding information indicating the recognition result of the recognition process (in which area the dynamic element D1 is located) and distance information derived based on the parallax information (distance information indicating the distance of the dynamic element D1 in the depth direction of the image) to the first image P1 or the second image P2.

[0087] [Effects of implementation methods]

[0088] As described above, in the image processing device 10 of the embodiment, in the static pixel value derivation process, the control unit 11 (static pixel value derivation unit 100) derives the static pixel value SS of the static pixel included in the image P0 based on the change of the pixel value S of each pixel of the image P0 over time. In the static image generation process, the control unit 11 (static image generation unit 103) generates a static image P10 representing the static pixel value SS of each pixel based on the static pixel value SS.

[0089] Specifically, the static pixel value derivation process includes a judgment process and a derivation process. In the judgment process, the control unit 11 (judgment unit 101) determines whether each pixel of the image P0 belongs to a dynamic pixel or a static pixel based on the change of the pixel value S of the pixel over time. In the derivation process, the control unit 11 (derivation unit 102) derives the static pixel value SS of each pixel of the image PO based on the pixel value S of the pixel at the past moment and the pixel value S of the pixel at the current moment when it is judged to belong to a static pixel through the judgment process. In the static image generation process, the control unit 11 (static image generation unit 103) generates a static image P10 representing the static pixel value SS of each pixel based on the static pixel value SS derived for each pixel of the image P0 through the derivation process.

[0090] In the above structure, a static image P10 can be generated using the pixel value S of the static pixel. In addition, the static pixel value SS can be derived based on the pixel value S of the pixel at the past moment and the pixel value S of the current moment (the pixel value S when it is determined to be a static pixel). In this way, the noise component contained in the static pixel value SS can be reduced, so a low-noise static image P10 can be generated. In this way, a low-noise static image P10 can be generated using the pixel value S of the static pixel.

[0091] In addition, in the image processing device 10 of the embodiment, during the export process, the control unit 11 (export unit 102) restores the static pixel value SS of each pixel of the image P0 to the initial value if the pixel is determined to be a dynamic pixel during the determination process.

[0092] It should be noted that if a pixel is determined to be a dynamic element, it is difficult to use the pixel value of the pixel to update the static pixel value SS. Therefore, when a pixel is determined to be a dynamic pixel, the static pixel value SS of the pixel can be appropriately initialized by restoring the static pixel value SS of the pixel to the initial value.

[0093] In addition, in the image processing device 10 of the embodiment, the storage unit 12 stores the static pixel value SS of each pixel of the image P0. In the derivation process, the control unit 11 (derivation unit 102) derives the static pixel value SS of each pixel of the image P0 based on "the static pixel value SS of the pixel stored in the storage unit 12" and "the pixel value S of the pixel (the pixel determined to be a static pixel in the determination process)", and updates the static pixel value SS of the pixel stored in the storage unit 12 to "the derived static pixel value SS".

[0094] In the above configuration, compared with the case where a plurality of pixel values ​​S (pixel values ​​S determined to be static pixels) used to derive static pixel values ​​SS are stored in the storage unit 12, the storage capacity required for storing pixel values ​​can be further reduced.

[0095] In the image processing device 10 of the embodiment, in the judgment process, the control unit 11 (judgment unit 101) judges that each pixel of the image P0 is a dynamic pixel when the amount of change of the pixel value S of the pixel over time exceeds the threshold, and judges that the pixel is a static pixel when the amount of change of the pixel value S of the pixel over time does not exceed the threshold. The threshold is set individually for each pixel of the image P0.

[0096] In the above configuration, since the threshold can be set individually for each pixel, the threshold can be set appropriately for each pixel. For example, since the threshold can be set for each pixel in consideration of optical shot noise, the robustness against ambient light fluctuations can be improved.

[0097] Furthermore, in the image processing device 10 according to the embodiment, the control unit 11 (determination unit 101 ) sets the threshold value set for each pixel of the image P0 to a value corresponding to the static pixel value SS of the pixel.

[0098] In the above configuration, by setting the threshold to a value corresponding to the static pixel value SS, a low-noise threshold can be set, thereby improving the accuracy of the determination process.

[0099] In addition, in the image processing device 10 of the embodiment, in the stereo matching process, the control unit 11 (stereo matching unit 104) performs stereo matching on pixels in the first image P1 that are judged to be dynamic pixels in the judgment process and pixels in the second image P2 that are judged to be dynamic pixels in the judgment process, thereby obtaining disparity information related to the dynamic elements shown in the first image P1 and the second image P2.

[0100] In the above configuration, in each of the first image P1 and the second image P2, dynamic pixels are used as the target of stereo matching, and static pixels are not used as the target of stereo matching, so that the processing load required for stereo matching can be reduced.

[0101] In the image processing device 10 of the embodiment, in the filling process, the control unit 11 (filling unit 106) fills the image portion PP inside the outline portion D2 of the dynamic element D1 shown in the characteristic image P3 based on the static image P10 generated by the static image generation process. The characteristic image P3 is generated by the characteristic image generation process. In the recognition process, the control unit 11 (recognition unit 107) recognizes the dynamic element D1 from the characteristic image (filling image P4) processed by the filling process.

[0102] In the above structure, by filling the characteristic image P3, recognition processing can be performed not only based on the contour portion D2 of the dynamic element D1 shown in the characteristic image P3, but also based on the image portion PP filled inside the contour portion D2. In this way, the accuracy of the recognition processing can be improved.

[0103] (Variation of the embodiment)

[0104] The image processing device 10 of the modification of the embodiment differs from the image processing device 10 of the embodiment in the derivation process (specifically, step S14). The other structures and processes of the image processing device 10 of the modification of the embodiment are the same as those of the image processing device 10 of the embodiment.

[0105] In step S14 of a variation of the implementation example, when the object pixel is determined to be a static pixel in step S13, the control unit 11 (derivation unit 102) derives the static pixel value SS of the object pixel based on the "static pixel value SS of the object pixel stored in the storage unit 12" and the "pixel value S of the object pixel", so that the greater the number of pixel values ​​S used to derive the "static pixel value SS of the object pixel stored in the storage unit 12", the more importance the "static pixel value SS of the object pixel stored in the storage unit 12" receives over the "pixel value S of the object pixel".

[0106] In this example, the control unit 11 assigns a weight to the "static pixel value SS of the object pixel stored in the storage unit 12", and averages (weighted average) the "static pixel value SS of the object pixel stored in the storage unit 12" and the "pixel value S of the object pixel" to which the weight is assigned, and sets the value obtained after the averaging (weighted average) as the new static pixel value SS of the object pixel, wherein the weight corresponds to the number of pixel values ​​S used to derive the "static pixel value SS of the object pixel stored in the storage unit 12".

[0107] Specifically, if the static pixel value SS derived from the object pixel of the Nth image P0 on the time axis is set to “SS (N) ”, and the static pixel value SS of the object pixel stored in the storage unit 12 (the static pixel value derived based on the object pixel of the N-1th image P0 on the time axis) is set to “SS (n-1) ", the new static pixel value SS (N) It is expressed as follows in Formula 3.

[0108] [Mathematical formula 3]

[0109]

[0110] [Effects of Modifications of Embodiment]

[0111] As described above, in the image processing device 10 of the variation of the embodiment, in the export process, the control unit 11 (export unit 102) exports the static pixel value SS of the image P0 for each pixel, based on "the static pixel value SS of the pixel stored in the storage unit 12" and "the pixel value S of the pixel (the pixel judged to be a static pixel in the judgment process)", so that the more pixel values ​​S used to export the static pixel value SS of the pixel stored in the storage unit 12, the more importance the static pixel value SS of the pixel stored in the storage unit 12 is given than the pixel value S of the pixel (the pixel judged to be a static pixel in the judgment process).

[0112] It should be noted that the more pixel values ​​S used to derive the static pixel value SS, the closer the static pixel value SS can be to the true value (the value not including noise), and the reliability of the static pixel value SS is improved. Therefore, by making the number of pixel values ​​S observed to derive the static pixel value SS stored in the storage unit 12 larger, the static pixel value SS stored in the storage unit 12 is given more importance, thereby making it easy to maintain the reliability of the newly derived static pixel value SS.

[0113] (Other embodiments)

[0114] In the above description, the image processing device 10 processes the first image P1 and the second image P2 acquired by the stereo camera (the first camera 21 and the second camera 22) as an example, but the invention is not limited thereto. For example, the image processing device 10 may also be configured to process an image acquired by a single camera. In this case, the control unit 11 may be configured to perform other processing (for example, other processing for acquiring distance information based on the image) instead of the stereo matching processing.

[0115] The above embodiments may be combined as appropriate and implemented. The above embodiments are essentially preferred examples, and are not intended to limit the technology disclosed herein, its application objects, or its scope of use.

[0116] -Industrial Applicability-

[0117] In summary, the technology disclosed here is useful as an image processing device and an image processing method.

[0118] -Explanation of symbols-

[0119] 10 Image processing device

[0120] 11. Control Unit

[0121] 12 Storage

[0122] 21 First Camera

[0123] 22 Second Camera

[0124] 100 Static pixel value export unit

[0125] 101 Judgment Department

[0126] 102 Export

[0127] 103 Still image generation unit

[0128] 104 Stereo Matching Unit

[0129] 105 Feature image generation unit

[0130] 106 Filling

[0131] 107 Identification Department

[0132] 108 Three-dimensional data generation department

[0133] P0 Image

[0134] P1 First Image

[0135] P2 Second image

[0136] P3 feature image

[0137] P4 Fill Image

[0138] P10 Still image.

Claims

1. An image processing device that processes images captured by a camera at predetermined intervals, characterized in that: The image processing device includes a control unit connected to the photographing device. The control unit includes a static pixel value deriving unit and a static image generating unit. The static pixel value deriving unit derives the static pixel value of the static pixel included in the image based on the change of the pixel value of each pixel of the image over time. The still image generation unit generates a still image representing the still pixel value for each pixel based on the still pixel value.

2. The image processing device according to claim 1, characterized in that: The static pixel value deriving unit includes a determining unit and a deriving unit. The determination unit determines, for each pixel of the image, whether the pixel belongs to a dynamic pixel or a static pixel based on the change of the pixel value of the pixel over time. The derivation unit derives, for each pixel of the image, a static pixel value of the pixel based on a pixel value of the pixel at a past time and a pixel value at a current time when the pixel is determined by the determination unit to be the static pixel. The still image generation unit generates a still image representing the still pixel value of each pixel based on the still pixel value derived by the derivation unit for each pixel of the image.

3. The image processing device according to claim 2, characterized in that: The deriving unit restores the static pixel value of each pixel of the image to an initial value when the determining unit determines that the image belongs to the dynamic pixel.

4. The image processing device according to claim 2 or 3, characterized in that: The image processing device includes a storage unit, the storage unit storing a static pixel value of each pixel of the image, For each pixel of the image, when the judgment unit determines that the pixel belongs to the static pixel, the export unit exports the static pixel value of the pixel based on the static pixel value of the pixel stored in the storage unit and the pixel value of the pixel judged to belong to the static pixel, and updates the static pixel value of the pixel stored in the storage unit to the exported static pixel value.

5. The image processing device according to claim 4, characterized in that: For each pixel of the image, when the judgment unit judges that the pixel belongs to the static pixel, the derivation unit derives the static pixel value of the pixel based on the static pixel value of the pixel stored in the storage unit and the pixel value of the pixel judged to belong to the static pixel, so that the more pixel values ​​used to derive the static pixel value of the pixel stored in the storage unit, the more importance the static pixel value of the pixel stored in the storage unit receives than the pixel value of the pixel judged to belong to the static pixel.

6. The image processing device according to any one of claims 2 to 5, characterized in that: The judging unit judges, for each pixel of the image, that the pixel belongs to the dynamic pixel when the amount of change of the pixel value of the pixel over time exceeds a threshold value, and judges that the pixel belongs to the static pixel when the amount of change of the pixel value of the pixel over time does not exceed the threshold value, The threshold is set individually for each pixel of the image.

7. The image processing device according to claim 6, characterized in that: The determination unit sets the threshold value set for each pixel of the image to a value corresponding to a static pixel value of the pixel.

8. The image processing device according to any one of claims 2 to 7, characterized in that: acquiring a first image and a second image having different viewpoints at each predetermined time. The determination unit performs processing based on the first image and the second image respectively, The deriving unit and the still image generating unit perform processing based on at least one of the first image and the second image, The control unit includes a stereo matching unit, a feature image generating unit, a filling unit and a recognition unit. The stereo matching unit performs stereo matching on pixels of the first image that are determined as the dynamic pixels by the determination unit and pixels of the second image that are determined as the dynamic pixels by the determination unit, thereby acquiring disparity information related to dynamic elements shown in the first image and the second image. The feature image generating unit generates a feature image representing a contour portion of the dynamic element based on the parallax information obtained by the stereo matching unit. The filling unit fills the image portion inside the outline portion of the dynamic element shown in the characteristic image based on the static image generated by the static image generating unit, wherein the characteristic image is generated by the characteristic image generating unit, The recognition unit recognizes the dynamic element from the feature image processed by the filling unit.

9. An image processing method, which processes images captured by a camera at predetermined intervals, characterized in that: The image processing method comprises: A step of deriving static pixel values ​​of static pixels contained in the image based on changes in the pixel value of each pixel of the image over time; and A step of generating a static image representing the static pixel value of each pixel based on the static pixel value.