Noise reduction method, filter circuit, and event-driven vision sensor

Abstract

This invention provides an event-driven vision sensor that detects changes in the intensity of incident light as events and outputs them as event signals, and a noise reduction method, a filter circuit, and an event-driven vision sensor that remove noise by not outputting event signals for events caused by noise. [Solution] In areas with a low event density, a first threshold is used, and in areas with a high event density, a second threshold greater than the first threshold is used. Event signals with an event density lower than the first and second thresholds are determined to be noise-related event signals, and these event signals are not output to remove the noise.

Description

[Technical Field]

[0001] The present invention relates to a noise reduction method for an event-driven vision sensor, a filter circuit used in an event-driven vision sensor, and an event-driven vision sensor. [Background technology]

[0002] Vision sensors are known as devices used in factories, surveillance cameras, etc., that capture a target workpiece with an image sensor such as a CCD to generate image data, and enable the determination of the state and positional deviation of the target workpiece based on this image data.

[0003] One form of such vision sensor is the event-driven vision sensor. In this event-driven vision sensor, an image is generated based on the change in the intensity of light incident on each pixel of the image sensor.

[0004] More specifically, in an event-driven vision sensor, when a change in the intensity of light incident on each pixel of the image sensor is detected, it is determined that an event has occurred and an event signal is output consisting of the coordinates of the pixel where the intensity change was detected, the time of detection, and the polarity of the change. The polarity of the change is such that, for example, if the intensity of light incident on the pixel changes in the direction of increasing, it is considered positive, and if the intensity of light incident on the pixel changes in the direction of decreasing, it is considered negative.

[0005] Furthermore, event-driven vision sensors generate a collection of events occurring at the same time based on the time information of the output event signals, generate an image signal for input to a display, and display the desired image by inputting it to the display.

[0006] The image displayed on the display is a dot-drawn image in which each event is displayed as a dot. In particular, in this dot-drawn image, by photographing the target work moving in front of the image sensor, dots are displayed at high density near the contour of the target work, resulting in an image that appears as if the contour line of the target work is drawn.

[0007] In such an event-driven vision sensor, no events should occur in areas where the target work does not exist. However, in reality, due to malfunctions of the pixels of the image sensor or the influence of reflected light from the target work, etc., events unrelated to the target work may occur. Such events are noise.

[0008] As a method of removing such noise, taking advantage of the fact that the number of events caused by noise is relatively small, the read frequency of the event signal is lowered (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0010] As described above, noise events not only appear in areas where the target work does not exist, but are also known to appear near a target work that moves relatively fast. If such noise appearing near the target work is to be detected under the same conditions as noise appearing in areas with a low event generation density, there is a problem that events that cannot be detected as noise are left behind.

[0011] When making a desired determination for the displayed image by determining an event that could not be removed as such noise as a true event, there was also a risk of causing a misjudgment.

[0012] An object of the present invention is to provide a noise removal method for an event-driven vision sensor, a filter circuit used for an event-driven vision sensor, and an event-driven vision sensor that can effectively remove not only noise that appears in an area where the event generation density is low but also noise that appears in an area where the event generation density is high in the vicinity of a target workpiece.

Means for Solving the Problem

[0013] The noise removal method for an event-driven vision sensor of the present invention is a noise removal method for an event-driven vision sensor that detects a change in the intensity of incident light as an event and outputs it as an event signal, and does not output an event signal of an event caused by noise to remove the noise. In particular, when the event generation density is smaller than the switching threshold value, the first threshold value is used, and when the event generation density is equal to or greater than the switching threshold value, the second threshold value greater than the first threshold value is used, and an event signal of an event having a generation density smaller than the first threshold value and the second threshold value is determined as an event signal of an event caused by noise.

[0014] Furthermore, the filter circuit for the event-driven vision sensor of the present invention is an event-driven vision sensor that detects changes in the intensity of incident light as events and outputs them as event signals. The filter circuit for the event-driven vision sensor identifies event signals caused by noise and removes noise by preventing the output of such event signals. In particular, it has a density measurement unit that measures the event occurrence density and a determination unit that determines whether the event signal of an event is an event signal caused by noise based on the occurrence density. The determination unit has a first threshold and a second threshold that is greater than the first threshold. When the occurrence density measured by the density measurement unit is less than the switching threshold, the first threshold is used. When the occurrence density measured by the density measurement unit is greater than or equal to the switching threshold, the second threshold is used. The determination unit is characterized by determining that event signals of events with occurrence densities smaller than the first threshold and the second threshold are event signals caused by noise.

[0015] Furthermore, the filter circuit of the event-driven vision sensor of the present invention is also characterized by being composed of a field-programmable gate array, and the density measurement unit is also characterized by using a field-programmable gate array block RAM.

[0016] Furthermore, the event-driven vision sensor of the present invention is an event-driven vision sensor that detects changes in the intensity of incident light as events and outputs them as event signals. It is an event-driven vision sensor that removes noise by identifying event signals of events caused by noise and preventing them from being output. In particular, it has a density measurement unit that measures the event occurrence density and a determination unit that determines whether the event signal of an event is an event signal of an event caused by noise based on the occurrence density. The determination unit has a first threshold and a second threshold that is greater than the first threshold. When the occurrence density measured by the density measurement unit is less than the switching threshold, the first threshold is used, and when the occurrence density measured by the density measurement unit is equal to or greater than the switching threshold, the second threshold is used. The determination unit is characterized by determining that the event signal of an event with an occurrence density less than the first threshold and the second threshold is an event signal caused by noise. [Effects of the Invention]

[0017] According to the present invention, the frequency of noise occurrence is estimated by evaluating the event occurrence density. By using a first threshold to remove noise appearing in areas with low event occurrence density and a second threshold to remove noise appearing in areas with high event occurrence density, events that are noise can be effectively removed. Therefore, the event-driven vision sensor of the present invention can make desired decisions using true event signals, making it less likely to cause false judgments. In particular, the event-driven vision sensor of the present invention can effectively remove noise by incorporating at least the filter circuit of the present invention. [Brief explanation of the drawing]

[0018] [Figure 1] This is an explanatory diagram of an event-driven vision sensor according to the present invention. [Figure 2] This is an explanatory diagram of the filter circuit according to the present invention. [Figure 3] This is a flowchart illustrating the operation of the filter circuit according to the present invention. [Modes for carrying out the invention]

[0019] The present invention provides a noise reduction method for an event-driven vision sensor, a filter circuit used in the event-driven vision sensor, and an event-driven vision sensor. By varying the threshold value used to determine noise depending on the event density, it is possible to remove not only noise generated in areas with low event density but also noise generated in areas with high event density. Therefore, it is possible to provide images that are less prone to misjudgment in determining whether a target workpiece is good or its position or orientation based on the image.

[0020] As shown in Figure 1, the event-driven vision sensor 1 comprises an image sensor 10 that captures a target workpiece, a processing unit 20 that processes the output signal from the image sensor 10, and a control device 30 that controls the processing unit 20 and the image sensor 10. The processing unit 20 generates an image signal, and by inputting this image signal to an appropriate display device 40, the image of the target workpiece can be displayed on the display device 40. In this embodiment, the display device 40 is for monitoring purposes and is not necessarily an essential device; instead of the display device 40, an appropriate judgment device (not shown) may be provided. Using this judgment device, a good product judgment or a determination of the position or orientation of the target workpiece may be made based on the image signal output from the processing unit 20, and the target workpiece may be manipulated as appropriate using a manipulator (not shown).

[0021] The image sensor 10 has multiple pixels and is capable of detecting the intensity of light incident on each pixel. The image sensor 10 can be a so-called CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. Each pixel of the image sensor 10 is arranged on a plane and has two-dimensional coordinate information.

[0022] The image sensor 10 is constantly receiving a timing signal for synchronization from the control device 30 via the first control signal line 31. Other control signals may also be input from the control device 30 to the image sensor 10 via the first control signal line 31.

[0023] The image sensor 10 determines that an event has occurred when it detects a change in the intensity of light incident on each pixel, and generates an event signal consisting of the coordinates of the pixel where the intensity change was detected, the time of detection, and the polarity of the change, and inputs this signal to the processing unit 20. In this embodiment, the polarity of the change is defined as positive when the intensity of light incident on the pixel increases, and negative when the intensity of light incident on the pixel decreases. Note that the positive and negative settings may be reversed. The timing at which the image sensor 10 detects a change in the intensity of light incident on each pixel is adjusted by a timing signal input from the control unit 30.

[0024] In this embodiment, the processing unit 20 consists of a filter circuit 21 that determines whether the event signal output from the image sensor 10 is an event signal based on an event caused by noise, and a drawing processing circuit 29 that generates an image signal from the event signal to be input to a display device 40 or the like.

[0025] The filter circuit 21 receives a timing signal for synchronization from the control device 30 via the second control signal line 32. Furthermore, the filter circuit 21 receives various signals from the control device 30 via the auxiliary control signal line 32'. The drawing processing circuit 29 receives a timing signal for synchronization from the control device 30 via the third control signal line 33. The second control signal line 32 and the third control signal line 33 may also transmit control signals other than timing signals, as well as signals for parameters required for each process.

[0026] The control device 30 performs various controls necessary for the event-driven vision sensor 1, including not only synchronous control of each device but also storage and output of necessary parameters such as threshold information, which will be described later.

[0027] The filter circuit 21, which is the main component of the present invention, will be described below.

[0028] As shown in Figure 1, the filter circuit 21 includes a density measurement unit 22 that measures the event occurrence density based on the event signal input from the image sensor 10, and a determination unit 23 that determines whether the input event signal is an event signal caused by noise based on this occurrence density.

[0029] As shown in Figure 2, the density measurement unit 22 consists of a first scheduler 22a, a first BRAM (Block Random Access Memory) 22b, a second BRAM 22c, a second scheduler 22d, an adder 22e, a first multiplexer 22f, and a second multiplexer 22g. The determination unit 23 is specifically composed of a comparator.

[0030] For the sake of explanation, let's assume that an event signal, represented as a vector (x1, y1, t1, p1), is input from the image sensor 10 to the density measurement unit 22. x1 and y1 are the two-dimensional coordinates of the pixel where the event was detected, t1 is the time the event was detected, and p1 is the polarity of the event.

[0031] The filter circuit 21 outputs the event signals (x1, y1, t1, p1) input from the image sensor 10 directly to the drawing processing circuit 29. At the same time, it outputs information to the drawing processing circuit 29 indicating whether the output event signals (x1, y1, t1, p1) are valid or invalid, that is, whether they are event signals caused by noise.

[0032] In the drawing processing circuit 29, if an input event signal is invalid, the event signal is discarded, and an image signal is generated using only the event signals that are determined to be valid.

[0033] As described above, the first scheduler 22a of the density measurement unit 22 outputs the event signals (x1, y1, t1, p1) input from the image sensor 10 directly to the drawing processing circuit 29, and also generates address information for the first BRAM 22b and the second BRAM 22c from the input coordinate information (x1, y1).

[0034] The first BRAM 22b and the second BRAM 22c are one of several block RAMs pre-installed within a field-programmable gate array (FPGA) when the filter circuit 21 is configured as such. A block RAM is a so-called line memory in which storage areas for one data item are arranged in a line, and it is used by converting the two-dimensional coordinates in the event signal into one-dimensional address information.

[0035] In particular, in this embodiment, the number of events detected in a 4x4 16 pixel area of ​​the image sensor 10 is stored in one memory area of ​​the block RAM. Note that it is not limited to 4x4, but could be, for example, 8x8 or 16x16. The first scheduler 22a performs address translation to identify the addresses of the first BRAM 22b and second BRAM 22c corresponding to the input event signals (x1, y1, t1, p1).

[0036] The first BRAM 22b of the density measurement unit 22 is a time management BRAM, and it is configured to overwrite the time information t1 of the input event signal (x1, y1, t1, p1) at a predetermined address in the first BRAM 22b. In particular, when overwriting the time information t1 of the event signal (x1, y1, t1, p1), the time information t before overwriting is used. past The time information t1 to be overwritten is then input to the second scheduler 22d.

[0037] The second BRAM 22c of the density measurement unit 22 is a density management BRAM that records the number of events that occurred at each pixel of the image sensor 10 corresponding to each address (16 pixels in this embodiment). The number of events stored at each address of this second BRAM 22c is expressed as "density" because it is the number of events that occurred within a predetermined area and within a predetermined time.

[0038] The second scheduler 22d of the density measurement unit 22 adjusts the timing for sending the event occurrence count information stored in the second BRAM 22c to the determination unit 23, and the timing for resetting the second BRAM 22c.

[0039] The second scheduler 22d receives not only the desired timing signal via the second control signal line 32, but also time threshold information T for resetting the second BRAM 22c. th The following is being input. Furthermore, the first scheduler 22a also receives the desired timing signal and time threshold information T. th I am typing this.

[0040] The adder 22e of the density measurement unit 22 adds "1" to the number of events read from a predetermined address in the second BRAM 22c.

[0041] The first multiplexer 22f and the second multiplexer 22g of the density measurement unit 22 output either "N+1" or "1" at a predetermined timing based on the control signal from the second scheduler 22d, as will be described later.

[0042] The first multiplexer 22f adjusts the timing of outputting event occurrence information, i.e., event occurrence density, to the determination unit 23.

[0043] The second multiplexer 22g overwrites the number of occurrences of the events added by the adder 22e at a predetermined address in the second BRAM 22c. Further, the second multiplexer 22g resets the second BRAM 22c by overwriting "1" at a predetermined address in the second BRAM 22c based on a control signal from the second scheduler 22d.

[0044] The determination unit 23 is a comparator as described above, and at the timing when the event generation density is input from the first multiplexer 22f, the switching threshold value information TH switch input from the control device 30 via the auxiliary control signal line 32', low the first density threshold value information TH high and the second density threshold value information TH

[0045] Based on these, the determination of the event signal (x1, y1, t1, p1) is performed. The determination is whether the event signal (x1, y1, t1, p1) is "valid" or "invalid", and the determination result is input to the drawing processing circuit 29. th The timing signal, the time threshold value information T switc the switching threshold value information TH low the first density threshold value information TH high and the second density threshold value information TH

[0046] In the filter circuit 21, an event signal (x1, y1, t1, p1) is input from the image sensor 10 to the first scheduler 22a (step S1).

[0047] The first scheduler 22a performs address translation processing to identify the addresses of the first BRAM 22b and the second BRAM 22c from the input event signals (x1, y1, t1, p1) (step S2). The first scheduler 22a also inputs the input event signals (x1, y1, t1, p1) to the drawing processing circuit 29.

[0048] Next, the filter circuit 21 overwrites the address of the first BRAM 22b identified in step S2 with the time information t1 of the event signal (x1, y1, t1, p1). At this time, the first BRAM 22b stores the time information t before it is overwritten. past Read the time information t before this overwrite. past The time information t1 is input to the second scheduler 22d (step S3).

[0049] In the second scheduler 22d, the time information t1 input from the first BRAM 22b and the time information t before overwriting are used. past The difference is calculated, and this difference value is input from the control device 30 as time threshold information T th The system determines whether the above conditions are met (Step S4).

[0050] Time information t1 and time information t before overwriting past The difference value is the time threshold information T th If the value is smaller (step S4: NO), the filter circuit 21 reads out the number of events N stored at the address of the second BRAM 22c identified in step S2 (step S5).

[0051] In the filter circuit 21, the number of events N read from the second BRAM 22c is incremented by the adder 22e (step S6), and the resulting number of events, which has become N+1, is input to the first multiplexer 22f and the second multiplexer 22g.

[0052] Furthermore, time information t1 and time information t before overwriting pastThe difference value is the time threshold information T th If the value is smaller (step S4: NO), the second scheduler 22d causes the first multiplexer 22f to output "1". Furthermore, the second scheduler 22d causes the second multiplexer 22g to output the incremented number of events N+1 input from the adder 22e.

[0053] In the filter circuit 21, the incremented number of events N+1 output from the second multiplexer 22g is overwritten at a predetermined address in the second BRAM 22c (step S7).

[0054] Meanwhile, the first multiplexer 22f inputs "1" to the comparator of the determination unit 23 (step S8).

[0055] In the comparator that receives a "1" input from the first multiplexer 22f, none of the threshold conditions described later are met, and therefore it outputs "invalid" (step S9). The filter circuit 21, having outputted "invalid," finishes one process and waits for the next event signal to be input.

[0056] In the process described above, the number of events is counted up in step S7, and the event density is measured by measuring the number of events that occurred in a predetermined area during a predetermined period. In other words, the number of events N stored at a predetermined address in the second BRAM22c is also the event density.

[0057] In step S4, the time information t1 and the time information t before overwriting are used. past The difference value is the time threshold information T th In the above case (step S4: YES), the filter circuit 21 reads out the number of events N stored at the address of the second BRAM 22c identified in step S2 (step S10).

[0058] In the filter circuit 21, the number of events N read from the second BRAM 22c is incremented by the adder 22e (step S11), and the resulting number of events, which has become N+1, is input to the first multiplexer 22f and the second multiplexer 22g.

[0059] Furthermore, time information t1 and time information t before overwriting past The difference value is the time threshold information T th In the above case (Step S4: YES), the second scheduler 22d causes the second multiplexer 22g to output "1", overwriting the predetermined address of the second BRAM 22c with "1" (Step S12). In other words, the predetermined address of the second BRAM 22c is reset.

[0060] Furthermore, the second scheduler 22d outputs the incremented number of events N+1 input from the adder 22e via the first multiplexer 22f and inputs it to the comparator of the determination unit 23 (step S13).

[0061] As described above, the comparator of the determination unit 23 receives switching threshold information TH via the auxiliary control signal line 32'. switc , threshold information for the first density TH low , and second density threshold information TH high These are input sequentially, and the event occurrence density "N+1" input from the first multiplexer 22f and the switching threshold information TH are entered. switc A comparison is being made with (Step S14).

[0062] The event occurrence density "N+1" is the switching threshold information TH switc If it is smaller than (step S14: NO), the threshold information for the first density TH is used as the first threshold. low Using this, the event occurrence density "N+1" and the threshold information TH for the first density are used. low A comparison is being made with (Step S15).

[0063] The event occurrence density "N+1" is determined by the threshold information TH for the first density.low If it is smaller than (Step S15: NO), the event of the event signal (x1, y1, t1, p1) is considered noise, and the comparator outputs "invalid" (Step S16).

[0064] The event occurrence density "N+1" is the switching threshold information TH switc If the above is true (Step S14: YES), the second density threshold information TH is used as the second threshold. high Using this, the event occurrence density "N+1" and the threshold information TH for the second density are used. high A comparison is being made with (Step S17). Note that the threshold information for the second density TH is also provided. high This is the threshold information TH for the first density. low The second threshold is a value greater than the first threshold.

[0065] The event occurrence density "N+1" is used as threshold information TH for the second density. high If it is smaller than (step S17: NO), the event of the event signal (x1, y1, t1, p1) is considered noise, and the comparator outputs "invalid" (step S16).

[0066] On the other hand, the event occurrence density "N+1" is the threshold information TH for the first density. low If the above is true, step S15: YES), and the event occurrence density "N+1" is the threshold information TH for the second density. high If the above is true, then in step S17:YES, the comparator outputs "valid" (step S18).

[0067] As described above, the drawing processing circuit 29 discards invalid event signals and generates image signals using only event signals that are determined to be valid.

[0068] Thus, in the filter circuit 21 of this embodiment, the event occurrence density is switched threshold information TH switc If it is smaller than, the threshold information TH for the first density lowUsing this, the event occurrence density is switched threshold information TH switc If the above is true, the threshold information for the second density TH high By using this method for noise detection, it is possible to effectively perform noise detection while taking into account the density of events.

[0069] This allows for proper removal of noise and identification of true events, enabling the system to perform the desired judgment on the displayed image and eliminate the risk of misjudgment.

[0070] Note that the switching threshold information TH switc , threshold information for the first density TH low , and second density threshold information TH high It is desirable to optimize the value by adjusting it while inputting appropriate values.

[0071] In this embodiment, the filter circuit 21 described above is constructed using a commercially available field-programmable gate array (FPGA). Therefore, by simply incorporating the filter circuit 21 constructed with an FPGA into an event-driven vision sensor, the noise reduction effect of the event-driven vision sensor can be improved, and the judgment accuracy of the event-driven vision sensor can be improved.

[0072] Moreover, the filter circuit 21 constructed with FPGA enables high-speed processing and provides a practical noise reduction method.

[0073] In particular, FPGAs generally have multiple block RAMs, so by adjusting the capacity to match these block RAMs, the filter circuit 21 can be incorporated into any event-driven vision sensor.

[0074] Alternatively, instead of incorporating it as an FPGA, the filter circuit 21 can be pre-built into the system circuit of the event-driven vision sensor, and building it as a dedicated circuit can enable faster operation.

[0075] Alternatively, a subroutine for the filter circuit 21 could be constructed as a programmatic data processing method to perform noise reduction. [Explanation of Symbols]

[0076] 1. Event-driven vision sensor 10 Image Sensors 20 Processing Units 21 Filter Circuit 22 Density measurement unit 22a First Scheduler 22b 1st BRAM 22c 2nd BRAM 22d Second Scheduler 22e Adder 22f 1st Multiplexer 22g Second Multiplexer 23 Judgment section 29 Drawing Processing Circuit 30 Control device 31. First control signal line 32 Second control signal line 32' Auxiliary control signal line 33 Third Control Signal Line 40 Display devices

Claims

1. In an event-driven vision sensor that detects changes in the intensity of incident light as an event and outputs it as an event signal, a noise reduction method for an event-driven vision sensor removes noise by not outputting event signals for events caused by noise, If the occurrence density of the aforementioned events is less than the switching threshold, the first threshold is used. If the occurrence density of the aforementioned events is greater than or equal to the switching threshold, a second threshold greater than the first threshold is used. A noise reduction method for an event-driven vision sensor, which determines that the event signals of events with an occurrence density smaller than the first threshold and the second threshold are event signals of events caused by noise.

2. In an event-driven vision sensor that detects changes in the intensity of incident light as an event and outputs it as an event signal, the filter circuit of the event-driven vision sensor removes noise by identifying the event signal of an event caused by noise and preventing the output of that event signal. A density measurement unit for measuring the occurrence density of the aforementioned events, A determination unit that determines whether the event signal of the event is the event signal of an event caused by the noise, based on the occurrence density of the event. It has, In the determination unit, It has a first threshold and a second threshold that is greater than the first threshold, If the generation density measured by the density measurement unit is less than the switching threshold, the first threshold is used. If the generation density measured by the density measurement unit is equal to or greater than the switching threshold, the second threshold is used. A filter circuit for an event-driven vision sensor that determines the event signal of an event with an occurrence density smaller than the first threshold and the second threshold as an event signal caused by the noise.

3. A filter circuit for an event-driven vision sensor according to claim 2, comprising a field-programmable gate array.

4. The event-driven vision sensor filter circuit according to claim 3, wherein the density measurement unit uses the block RAM of the field-programmable gate array.

5. In an event-driven vision sensor that detects changes in the intensity of incident light as events and outputs them as event signals, the event-driven vision sensor removes noise by identifying event signals caused by noise and preventing the output of those event signals. A density measurement unit for measuring the occurrence density of the aforementioned events, A determination unit that determines whether the event signal of the event is the event signal of an event caused by the noise, based on the occurrence density of the event. It has, In the determination unit, It has a first threshold and a second threshold that is greater than the first threshold, If the generation density measured by the density measurement unit is less than the switching threshold, the first threshold is used. If the generation density measured by the density measurement unit is equal to or greater than the switching threshold, the second threshold is used. An event-driven vision sensor that determines the event signal of an event with an occurrence density smaller than the first threshold and the second threshold as an event signal caused by noise.

Images