Image inpainting apparatus, image inpainting method, and image inpainting program

Abstract

The image restoration device (3) of the present application is provided with an initialization block (100) that initializes the luminance value of each pixel coordinate to an intermediate value in a luminance arrangement list (L) that stores a pair of polarity values and an arbitrary value among the intermediate values as the luminance value of each pixel coordinate. In addition, the image restoration device (3) is provided with an update block (120) that updates the initialized luminance arrangement list (L) according to the pixel coordinate and the polarity value of each event, and an output block (160) that outputs the luminance arrangement list (L) updated by the update block (120) during the entire shooting period as a binary image (B). By the update performed by the update block (120), the luminance value of the trigger coordinate of the event triggered in the luminance arrangement list (L) is overwritten by the polarity value of the event. Furthermore, by the update, the luminance value of the non-trigger coordinate other than the trigger coordinate in the luminance arrangement list (L) is maintained.

Description

Technical Field

[0001] This disclosure relates to image inpainting techniques for repairing binary images based on output from an event camera. Background Technology

[0002] As optical cameras that mimic the human perception system, event cameras (or simply event cameras) are known for their excellent temporal resolution. An event camera optically captures a subject and outputs the pixel coordinates and polarity value of the event in at least one camera pixel, in association with the triggering time of an event that causes a change in brightness.

[0003] For example, a technique for restoring a brightness image based on the output of such an event camera is disclosed in Non-Patent Document 1, etc. In the technique disclosed in Non-Patent Document 1, the optical flow and brightness of the subject moving relative to the event camera are estimated simultaneously by optimizing the cost function, thereby restoring the brightness image.

[0004] Existing technical documents

[0005] Non-patent literature

[0006] Non-patent literature 1: P.Bardow,AJDavison,and S.Leutenegger.SimultaneousOptical Flow and Intensity EstimationFrom an Event Camera.The IEEE Conference on ComputerVision and Pattern Recognition

[0007] (CVPR), pp. 884-892, 2016. Summary of the Invention

[0008] The technical problem to be solved by the present invention

[0009] However, the technique disclosed in Non-Patent Document 1, which simultaneously estimates optical flow and brightness, is limited to the edge portion of the subject's photographed surface where camera pixels can sense brightness changes through relative motion. Therefore, if brightness estimation is extended to the entire photographed surface of the subject to repair the brightness image, the computational processing becomes complex, thus increasing the computational load.

[0010] The technical problem this disclosure aims to solve is to provide an image restoration apparatus that reduces the computational load required for image restoration. Another technical problem this disclosure aims to solve is to provide an image restoration method that reduces the computational load required for image restoration. A further technical problem this disclosure aims to solve is to provide an image restoration program that reduces the computational load required for image restoration.

[0011] Technical means for solving technical problems

[0012] The technical means of this disclosure for solving the technical problem will be described below. Furthermore, the reference numerals in parentheses in the claims and this section indicate the correspondence between the specific technical means described in the embodiments which will be explained in detail later, and are not intended to limit the technical scope of this disclosure.

[0013] The first aspect of this disclosure provides an image restoration apparatus (3) that restores a binary image (B) based on the pixel coordinates (x, y) and polarity value (p) of an event, which is output from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, in association with the triggering time (t) of an event in which a brightness change occurs in at least one camera pixel. The apparatus comprises:

[0014] The initialization unit (100) initializes the brightness value of each pixel coordinate to the intermediate value in a brightness sorting list (L) that stores any one of a pair of polarity values ​​and their intermediate values ​​as the brightness value of each pixel coordinate.

[0015] The update unit (120) updates the brightness sort list initialized by the initialization unit according to the pixel coordinates and polarity values ​​of each event; and

[0016] The output unit (160) outputs a binary image by updating a brightness sorting list throughout the shooting process via the update unit.

[0017] In the update process, the brightness value of the trigger coordinate, which is the pixel coordinate that is the trigger of the event in the brightness arrangement list, is overwritten using the polarity value of the event. On the other hand, the brightness value of the non-trigger coordinate, which is the pixel coordinate other than the trigger coordinate, is maintained in the brightness arrangement list.

[0018] A second aspect of this disclosure provides an image restoration method, executed by a processor (12), to restore a binary image (B) based on the pixel coordinates (x, y) and polarity value (p) of an event, output in association with the trigger time (t) of an event in which a brightness change occurs in at least one camera pixel, from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, and the event itself. The method comprises:

[0019] In the initialization process (S101), the brightness value of each pixel coordinate is initialized to the intermediate value in the brightness sorting list (L) that stores any one of the polarity values ​​and their intermediate values ​​as the brightness value of each pixel coordinate.

[0020] The update process (S102, S202) updates the brightness sort list initialized by the initialization process based on the pixel coordinates and polarity values ​​of each event; and

[0021] The output process (S104) outputs a binary image by updating the brightness arrangement list throughout the shooting process.

[0022] In the update process, the brightness value of the trigger coordinate of the pixel coordinate that is the trigger of the event in the brightness arrangement list is overwritten using the polarity value of the event. On the other hand, the brightness value of the non-trigger coordinate that is the pixel coordinate other than the trigger coordinate is maintained in the brightness arrangement list.

[0023] A third aspect of this disclosure provides an image restoration program comprising a command stored in a storage medium (10) and executed by a processor (12) for restoring a binary image (B) based on pixel coordinates (x, y) and polarity value (p) of an event, output from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, in association with a trigger time (t) of an event involving a brightness change in at least one camera pixel, wherein the command comprises:

[0024] In the initialization process (S101), the brightness value of each pixel coordinate is initialized to the intermediate value in the brightness sorting list (L) that stores any one of the polarity values ​​and their intermediate values ​​as the brightness value of each pixel coordinate.

[0025] The update process (S102, S202) updates the brightness sort list initialized by the initialization process based on the pixel coordinates and polarity values ​​of each event; and

[0026] The output process (S104) outputs a binary image by updating the brightness arrangement list throughout the shooting process.

[0027] In the update process, the brightness value of the trigger coordinate of the pixel coordinate that is the trigger of the event in the brightness arrangement list is overwritten using the polarity value of the event. On the other hand, the brightness value of the non-trigger coordinate that is the pixel coordinate other than the trigger coordinate is maintained in the brightness arrangement list.

[0028] According to these first to third methods, for a brightness arrangement list where the brightness values ​​of each pixel coordinate are initialized to intermediate values, in the update corresponding to the pixel coordinates and polarity values ​​of each event, the brightness value of the trigger coordinate is overwritten using the polarity value of the event. At this time, the polarity value overwritten in the brightness value can represent the actual brightness of the portion of the photographed surface of the black-and-white subject where relative movement causes the camera pixel to trigger the event. Furthermore, according to the first to third methods, in the update of the initialized brightness arrangement list, this overwriting operation is performed on one hand, while the brightness values ​​of non-trigger coordinates other than the trigger coordinates are maintained on the other hand. As a result, the polarity value maintained after the polarity value overwriting operation at the non-trigger coordinates switched from the trigger coordinate can represent the actual brightness of the portion of the photographed surface that does not trigger an event even with relative movement.

[0029] Therefore, through the first to third methods, a brightness arrangement list updated throughout the entire shooting period can be output as a binary image that stores the polarity value corresponding to the actual brightness at each pixel coordinate corresponding to the photographed surface. Based on the above, the binary image can be repaired through a simple computational process called overwrite operation, thus reducing the computational load required for image restoration. Attached Figure Description

[0030] In the attached image:

[0031] Figure 1 This is a block diagram illustrating the overall structure of the image system according to the first embodiment.

[0032] Figure 2 This is a block diagram showing the detailed structure of the image restoration apparatus in the image system according to the second embodiment.

[0033] Figure 3 This is a perspective view used to schematically illustrate the relative motion relationship between the event camera and the black-and-white subject in the first embodiment.

[0034] Figure 4 This is a top view showing the subject of the first embodiment.

[0035] Figure 5 This is a schematic diagram used to illustrate the event data of the first embodiment.

[0036] Figure 6 This is a schematic diagram illustrating the brightness arrangement list of the first embodiment.

[0037] Figure 7 This is a schematic diagram illustrating the initialization process of the first embodiment.

[0038] Figure 8 This is a schematic diagram illustrating the update process of the first embodiment.

[0039] Figure 9 This is a schematic diagram illustrating the first sub-correction step of the first embodiment.

[0040] Figure 10 This is a schematic diagram illustrating the second sub-correction step of the first embodiment.

[0041] Figure 11 This is a schematic diagram illustrating a binary image of the first embodiment.

[0042] Figure 12 This is a flowchart illustrating the image restoration method of the first embodiment.

[0043] Figure 13 This is a flowchart illustrating the image restoration method of the second embodiment. Detailed Implementation

[0044] Hereinafter, several embodiments will be described with reference to the accompanying drawings. Furthermore, there are instances where repeated descriptions are omitted by assigning the same reference numerals to corresponding structural elements in each embodiment. When only a portion of the structure is described in each embodiment, the structures of other previously described embodiments can be applied to the remaining parts of that structure. Moreover, not only can the structures explicitly shown in the descriptions of each embodiment be combined, but even without explicit description, the structures of multiple embodiments can be partially combined with each other, provided there are no particular obstacles to combination.

[0045] (First Implementation)

[0046] like Figure 1 , Figure 2 As shown, the image system 1 of the first embodiment is configured to include an event-based camera 2 and an image restoration device 3. In the image system 1, based on the output from the event camera 2, which captured a black-and-white subject 4 at two gray levels, the image restoration device 3 restores a binary image B, which is a binary image of two gray levels. Assume that in the binary image B restored by the image restoration device 3, the brightness value (i.e., gray value) representing the black part of the black-and-white subject 4 is "0", and the brightness value representing the white part of the subject 4 is "1".

[0047] The event camera 2 is configured to establish a specific relative motion relationship between itself and the black-and-white subject 4, which is the subject of the photograph. This specific relative motion relationship refers to a so-called translational relationship, where the direction of the light flow generated during the shooting period, due to the relative motion with respect to the event camera 2, is the same across the entire area of ​​the photographed surface 40 of the black-and-white subject 4 captured by the event camera 2. An example of this translational relationship is when the planar black-and-white subject 4 moves linearly relative to the fixed-position event camera 2 only in a direction M orthogonal to the camera axis. Figure 3 Therefore, in this embodiment, a translational relationship is envisioned, namely, the black and white subject 4 as a planar object, such as... Figure 4 The QR code shown, depicted in two shades of gray (white and black), is printed or affixed to factory products and moves linearly via a conveyor belt, and is captured by a fixed-position event camera 2. Furthermore, Figure 4 The QR code shown is a QR code (a registered trademark of DENSO WAVE Co., Ltd.), but it could also be a barcode, etc.

[0048] like Figure 3 As shown, the event camera 2 captures the photographed surface 40 of a black-and-white subject 4, which is in relative motion during a specified shooting period, within the camera's field of view throughout the entire shooting period. The event camera 2 senses incident light from within the camera's field of view using multiple camera pixels arranged in a two-dimensional array in the horizontal and vertical directions. In the event camera 2, the brightness changes of the incident light sensed by each camera pixel are monitored sequentially during the shooting period. As a result, if the amount of brightness change in at least one camera pixel, either in an increasing or decreasing direction, is greater than or equal to a logarithmic threshold, an event is considered triggered, and event data D is output from the event camera 2. Figure 5 As shown, event data D is generated as a dataset used to output the pixel coordinates x, y, and polarity value p of the event in association with the event's trigger time t. Furthermore, in Figure 5 And will be explained later. Figures 6-8 In the diagram, X and Y represent the maximum values ​​of the x and y coordinates, respectively.

[0049] Pixel coordinates x and y define the two-dimensional position of camera pixels where the brightness change exceeds a threshold. The polarity value p defines the direction of increase or decrease in brightness change exceeding the threshold using a binary value. Specifically, in this embodiment, the polarity value p representing the direction of brightness increase from black to white is defined as "1," while the polarity value p representing the direction of brightness decrease from white to black is defined as "0." Thus, in the event data D, a pair of polarity values ​​p represents the direction of increase or decrease in brightness change at the pixel coordinates x and y that triggered the event.

[0050] Figure 1 , Figure 2 The image restoration device 3 shown is connected to the event camera 2, for example, via at least one of a LAN (Local Area Network), a wiring harness, and an internal bus. Figure 1 As shown, the image restoration apparatus 3 is a computer specifically configured to perform image restoration, comprising at least one memory 10 and one processor 12. The memory 10 is a storage medium capable of storing programs and data non-transitorily, such as semiconductor memory, magnetic media, and optical media, which can be read by a computer. The processor 12 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer) CPU, as its core.

[0051] The processor 12 executes multiple commands contained in the image restoration program stored in the memory 10. Thus, the image restoration apparatus 3 constructs multiple functional blocks for restoring the binary image B based on the event data D output from the event camera 2. In this way, in the image restoration apparatus 3, multiple functional blocks are constructed by the processor 12 executing sequentially or in parallel multiple commands of the image restoration program stored in the memory 10 for restoring the binary image B. Figure 2 As shown, the multiple functional blocks constructed by the image restoration device 3 include an initialization block 100, an update block 120, a correction block 140, and an output block 160.

[0052] Initialization block 100 prepares the brightness sorting list L in the specified data storage area in memory 10. For example... Figure 6 As shown, the brightness sorting list L is prepared as a dataset that associates brightness values ​​I with timestamps T, according to the x, y coordinates of each pixel in the two-dimensional coordinate system corresponding to the entire camera pixels of the event camera 2. The brightness value I stored in the brightness sorting list L is any one of the values ​​of a pair of polarity values ​​p corresponding to two grayscale brightness values ​​of the binary image B, and their intermediate value m. The intermediate value m is defined as falling within the range between the two values ​​of the pair of polarity values ​​p. In particular, in this embodiment, the intermediate value m is defined as the median "0.5" between the polarity value p (1) corresponding to the white brightness value of the binary image B in the direction of brightness increase, and the polarity value p (0) corresponding to the black brightness value of the image B in the direction of brightness decrease.

[0053] like Figure 7As shown, initialization block 100 initializes the brightness values ​​I of all pixel coordinates x and y in the brightness arrangement list L to the intermediate value m. At the same time, initialization block 100 initializes the timestamps T of all pixel coordinates x and y in the brightness arrangement list L to "0" as the reset value. These initialization procedures are implemented by initialization block 100 before or during the start of the shooting period of event camera 2.

[0054] Figure 2 The update block 120 shown retrieves event data D for each event during the shooting period from the event camera 2. The update block 120 can also retrieve the event data D output by the event camera 2 at each trigger time t, individually at that time t. Alternatively, the update block 120 can retrieve the event data D output by the event camera 2 at each trigger time t and buffered in the memory 10 for a set period, such as during the shooting period, after that set period.

[0055] Update block 120 updates the luminance sort list L based on the trigger time t, pixel coordinates x, y, and polarity value p contained in the acquired event data D. Update block 120 may also update the luminance sort list L at each trigger time t of the acquired event data D. Update block 120 may also update the luminance sort list L each time it extracts data from the buffered event data D for each trigger time t. In the following, the update procedure in any case will also be described in detail as a procedure for each event.

[0056] like Figure 8 As shown, the update block 120 of the update process sets the pixel coordinates x, y of the brightness arrangement list L corresponding to the pixel coordinates x, y of the trigger time t in the event data D to the trigger coordinates xf, yf at the trigger time t of the trigger event. At the same time, the update block 120 sets all pixel coordinates x, y in the brightness arrangement list L except for the trigger coordinates xf, yf to the non-trigger coordinates xn, yn at the trigger time t.

[0057] The update block 120 of the update process uses the polarity value p of the event at trigger time t in the event data D to overwrite the brightness value I of the trigger coordinates xf and yf at trigger time t in the brightness arrangement list L. At the same time, the update block 120 uses the trigger time t of the event in the event data D to overwrite the timestamp T of the trigger coordinates xf and yf at trigger time t in the brightness arrangement list L. At this time, the update block 120 of this embodiment takes the start time of the shooting period as the base "0" second, replaces the trigger time t with the elapsed time from that start time, and saves it as the timestamp T.

[0058] While performing these overwrites, the update block 120 of the update process maintains the brightness value I of the coordinates xn, yn at the non-trigger coordinates at the trigger time t in the brightness arrangement list L, where the time difference between the trigger time t and the timestamp T is within the allowed time range, as the saved value at the trigger time t. As a result, the brightness value I of the non-trigger coordinates xn, yn that switch from the trigger coordinates xf, yf in the brightness arrangement list L is maintained within the allowed time range from the trigger time t of the coordinates xf, yf before the switch.

[0059] However, in the brightness arrangement list L, for the non-trigger coordinates xn, yn at the trigger time t, the brightness value I of the coordinates where the time difference between the trigger time t and the timestamp T is outside the allowed time range, the update block 120 returns the intermediate value m through an overwrite operation. As a result, the brightness value I of the non-trigger coordinates xn, yn that switch from the trigger coordinates xf, yf in the brightness arrangement list L is outside the allowed time range from the trigger time t of the coordinates xf, yf before the switch, and is overwritten by the intermediate value m.

[0060] Furthermore, the allowable time range for switching the update process based on the time difference can be set to either an upper limit of the time difference for maintaining the brightness value I, or a lower limit of the time difference for overwriting the brightness value I with the intermediate value m.

[0061] Figure 2 The correction block 140 shown corrects the brightness arrangement list L that has been updated by the update block 120 throughout the entire shooting period. That is, the correction block 140 corrects the brightness arrangement list L that has been updated based on the event data D of all events during the shooting period. In the following, the correction process of the correction block 140 will be described in detail as a process for the brightness arrangement list L after all updates.

[0062] The correction block 140 of the correction process sets the entire pixel coordinates x and y of the intermediate value m stored in the brightness arrangement list L after all updates to intermediate coordinates xm and ym. Under this setting, the update block 120 performs a first sub-correction process and a second sub-correction process to correct the brightness value I of the intermediate coordinates xm and ym in the brightness arrangement list L from the intermediate value m to an appropriate value. The first sub-correction process will be explained below. Figure 9 The entire brightness ranking list L is shown using an image. On the other hand, to explain the second sub-correction process, Figure 10 A portion of the brightness ranking list L is shown in image form.

[0063] like Figure 9As shown, the correction block 140 of the first sub-correction process uses the starting coordinates xs and ys to extract the intermediate coordinates xm and ym of the edge portion constituting the brightness arrangement list L. At this time, the correction block 140 sets at least one of the coordinates x and y in the horizontal and vertical directions to the minimum or maximum value. Figure 9 The intermediate coordinates xm and ym (in the example where both are the minimum values) are set as the starting coordinates xs and ys. The correction block 140 of the first sub-correction process overwrites the brightness values ​​I of the coordinates xm and ym in the brightness arrangement list L that are directly continuous with the starting coordinates xs and ys or indirectly continuous with the starting coordinates xs and ys, and the brightness values ​​I of the coordinates xs and ys, using a polarity value p of a specific side. Here, when the black and white subject 4 is a QR code, the value "1" corresponding to the white brightness value of the binary image B, which represents the direction of brightness increase, is overwritten as a polarity value p of a specific side on the starting coordinates xs and ys and their continuous coordinates. Furthermore, the process described in this disclosure, "In the update of the update section, the brightness value of the trigger coordinate of the pixel coordinate that is the trigger event in the brightness arrangement list is overwritten using the polarity value of the event, while maintaining the brightness value of the non-trigger coordinate that is the pixel coordinate other than the trigger coordinate in the brightness arrangement list," is performed through this first sub-correction process. From this point of view, the first sub-correction process is the main process.

[0064] like Figure 10 As shown, the correction block 140 of the second sub-correction process sets the relative motion direction M of the black and white subject 4 relative to the event camera 2 as the search direction S. At this time, the relative motion direction M is calculated, for example, based on the output value from the accelerometer sensor, and set as the search direction S. The correction block 140 of the second sub-correction process searches for the coordinates of interest xc, yc that are immediately adjacent to the intermediate value m in the search direction S, even after the first sub-correction process, from the intermediate coordinates xm, ym in the brightness arrangement list L. The correction block 140 of the second sub-correction process overwrites the brightness value I of the remaining coordinates of the intermediate value m in the brightness arrangement list L with the polarity value p of the coordinates of interest xc, yc. As a result, in the brightness value I of the remaining coordinates, the value of the subject 4 relative to the intermediate value m is... Figure 10 The polarity value p of the coordinates of interest xc and yc is copied in the opposite direction O of the search direction S. In this way, by adding a second sub-correction step, the polarity value p can be determined without waiting for the entire event data D to be read through the main first sub-correction step.

[0065] Figure 2 The output block 160 shown outputs a binary image B, which is updated not only by update block 120 throughout the shooting period but also by correction block 140, refining the brightness arrangement list L. That is, output block 160 outputs a binary image B. Figure 11The binary image B shown in the output is repaired by updating the event data D at all trigger times t during the shooting period. The intermediate coordinates xm and ym are then corrected to the latest brightness sort list L of any polarity value p.

[0066] Based on the description so far, in the first embodiment, the initialization block 100 is equivalent to an "initialization unit" and the update block 120 is equivalent to an "update unit". Furthermore, in the first embodiment, the correction block 140 is equivalent to a "correction unit" and the output block 160 is equivalent to an "output unit".

[0067] according to Figure 12 The image restoration process of the image restoration device 3, which uses the cooperation of blocks 100, 120, 140, and 160 to restore the binary image B, will be described. Furthermore, this image restoration process begins before or concurrently with the start of shooting by the event camera 2. In this image restoration process, "S" refers to the multiple steps of the process executed by multiple commands included in the image restoration procedure.

[0068] In S101, initialization block 100 initializes the brightness arrangement list L prepared in memory 10. At this time, initialization block 100 initializes the brightness value I and the timestamp T of each pixel coordinate x and y in the brightness arrangement list L to the intermediate value m and the reset value, respectively.

[0069] In S102, following S101, update block 120 updates the brightness arrangement list L initialized in S101 based on the event data D of each event. At this time, as an update procedure for each event, update block 120 overwrites the brightness values ​​I and timestamp T of the trigger coordinates xf and yf in the brightness arrangement list L using the event polarity value p and the trigger time t, respectively. Simultaneously, as an update procedure for each event, update block 120 overwrites the brightness values ​​I of the non-trigger coordinates xn and yn in the brightness arrangement list L where the time difference between the trigger time t and the timestamp T is outside the allowed time range using an intermediate value m. As a result, the brightness values ​​I of the non-trigger coordinates xn and yn that switch from the trigger coordinates xf and yf in the brightness arrangement list L are overwritten by the intermediate value m, falling outside the allowed time range from the trigger time t of the coordinates xf and yf before the switch.

[0070] Furthermore, while performing these overwrite operations, as an update procedure for each event, update block 120 of S102 maintains the brightness value I of the coordinates xn, yn in the brightness arrangement list L where the time difference between the trigger time t and the timestamp T is within the allowed time range. As a result, the brightness value I of the non-trigger coordinates xn, yn that switch from the trigger coordinates xf, yf in the brightness arrangement list L is maintained within the allowed time range from the trigger time t of the coordinates xf, yf before the switch.

[0071] In S103, following S102, correction block 140 corrects the brightness arrangement list L updated by update block 120 based on event data D of all events during the entire shooting period. At this time, as a first sub-correction step, correction block 140 overwrites the brightness value I of the starting coordinates xs, ys of the edge portion of the intermediate coordinates xm, ym that store the intermediate value m in the brightness arrangement list L after the entire update, and the brightness value I of the coordinates continuous with xs, ys, using a polarity value p on a specific side. As described above, this first sub-correction step is the main step. Furthermore, as a second sub-correction step, correction block 140 overwrites the brightness value I of the remaining coordinates (xc, yc) in the brightness arrangement list L after the first sub-correction step, which retain the intermediate value m, with the polarity value p of the interest coordinates xc, yc that are adjacent to the remaining coordinates xm, ym in the search direction S. At this time, the search direction S is set to the relative motion direction M of the black-and-white subject 4 relative to the event camera 2. The processing speed can be increased through this second sub-correction process.

[0072] In S104, following S103, the output block 160 outputs the binary image B, which is then updated by the brightness arrangement list L via correction block 140 after being updated throughout the shooting process via S102. Based on the above, the image restoration process for one shooting period ends.

[0073] Based on the description so far, in the first embodiment, S101 corresponds to the "initialization process" and S102 corresponds to the "update process". Furthermore, in the first embodiment, S103 corresponds to the "correction process" and S104 corresponds to the "output process".

[0074] (Effects)

[0075] The effects of the first embodiment described above will be explained below.

[0076] According to the first embodiment, for each pixel coordinate x, y, the brightness value I is initialized to a brightness arrangement list L with an intermediate value m. In the update corresponding to the pixel coordinates x, y and polarity value p of each event, the brightness value I of the trigger coordinates xf, yf is overwritten with the polarity value p of the event. At this time, the polarity value p overwritten on the brightness value I can represent the actual brightness of the trigger portion (i.e., the brightness change portion that is given a brightness change) of the photographed surface 40 of the black and white subject 4, where the camera pixel triggers the event by relative movement. In addition, according to the first embodiment, in the update of the initialized brightness arrangement list L, this overwriting operation is performed on the one hand, while the brightness value I of the non-trigger coordinates xn, yn, other than the trigger coordinates xf, yf, is maintained on the other hand. As a result, the polarity value p maintained at the non-trigger coordinates xn, yn, which are switched from the trigger coordinates xf, yf, after the overwriting operation of the polarity value p, can represent the actual brightness of the non-trigger portion of the photographed surface 40 that does not trigger the event even by relative movement.

[0077] Based on these, the brightness arrangement list L updated throughout the shooting process using the first embodiment can be output as a binary image B, where each pixel coordinate x, y corresponding to the photographed surface 40 stores a polarity value p corresponding to the actual brightness. Based on the above, the binary image B can be repaired through a simple computational process called overwrite operation, thus reducing the computational load required for image restoration.

[0078] According to the first embodiment, in the brightness arrangement list L updated throughout the shooting period, the brightness value I of the intermediate coordinates xm, ym, which stores the intermediate value m, is overwritten with the polarity value p of the interest coordinates xc, yc, which are adjacent to the intermediate coordinates xm, ym in the search direction S. At this time, by setting the search direction to the relative motion direction M of the black and white subject 4 relative to the event camera 2, the polarity value p, which represents the higher probability of the actual brightness of the subject surface 40 in that direction M, can be repaired by replacing the intermediate value m. Therefore, poor repair of the binary image B caused by the remnant of the intermediate value m can be suppressed.

[0079] According to the first embodiment, the brightness value I of the non-trigger coordinates xn, yn switched from the trigger coordinates xf, yf in the brightness arrangement list L remains at the polarity value p of the original coordinates xf, yf within an allowed time range from the trigger time t of the original coordinates xf, yf. As a result, the polarity value p maintained within the allowed time range in the non-trigger coordinates xn, yn switched from the trigger coordinates xf, yf can represent the actual brightness of the non-trigger portion of the photographed surface 40 that no longer triggers the event due to relative movement after the event trigger. Therefore, by performing simple computational processing, the restoration of the binary image B at the non-trigger portion, which previously required estimation of optical flow, can be achieved, reducing the computational load.

[0080] According to the first embodiment, in the brightness arrangement list L, the brightness value I of the non-trigger coordinates xn, yn switched from the trigger coordinates xf, yf is overwritten to an initial intermediate value m outside the allowed time range from the trigger time t of the coordinates xf, yf before the switch. As a result, in the case of an event triggered by noise at the trigger coordinates xf, yf, even if the brightness value I is overwritten by an incorrect polarity value p, no subsequent event triggering will occur until outside the allowed time range, thus the brightness value I can return to the normal intermediate value m. Therefore, poor restoration of the binary image B caused by noise can be suppressed.

[0081] According to the first embodiment, the direction of the optical flow that occurs during the shooting period due to the relative motion with respect to the event camera 2 is the same throughout the entire area of ​​the photographed surface 40 captured by the event camera 2. Therefore, after the overwrite operation of the polarity value p, the polarity value p of the non-trigger coordinates xn, yn switched from the trigger coordinates xf, yf is unlikely to deviate from the actual brightness of the non-trigger portion. Thus, the binary image B can be repaired through simple computational processing.

[0082] In the first embodiment of image restoration using a QR code that moves relative to the event camera 2 during the shooting period as a black-and-white subject 4, the polarity value p, which is switched from the trigger coordinates xf, yf to the non-trigger coordinates xn, yn after the overwrite operation of the polarity value p, can accurately represent the actual brightness of the non-trigger portion. Therefore, with simple calculation processing, it is also possible to restore the binary image B with high precision.

[0083] Next, the first embodiment described above will be explained again from another perspective. First, the black and white photographic object 4, which is the object to be read in this disclosure, is not limited to a QR code (a registered trademark of DENSO WAVE Co., Ltd.) as described above, but can also be other types of QR codes, and further, can also be a barcode as a one-dimensional code. It is usually printed in white and black, but it can also have color as long as it is a light color and a dark color. Therefore, the white and black used in the following description refer to light colors and dark colors.

[0084] like Figures 1-3 As shown, in the case of a QR code, there are fine quadrilateral patterns 4a at the three corners of the code. Inside the code, similar to the white quadrilateral cells, are arranged black quadrilateral cells, and information is determined based on this cell arrangement. The width of the outermost black quadrilateral of the fine pattern 4a is the same as the width of the cell, and the width of the inner white quadrilateral is also the same as the width of the cell. The width of the central black quadrilateral is three times the width of the cell. In any case, a printed QR code is a combination of white or black quadrilaterals.

[0085] Figure 4 The image shown is of a QR code restored by the image restoration device 3. In its printed state, the QR code did not... Figure 4 That kind of distortion. However, QR codes determine the brightness (white, black) of the center of each cell, so even if it's like... Figure 4 The repaired QR code can also be read.

[0086] Additionally, Event Camera 2, such as Figure 9 The brightness arrangement list L shows a number of camera pixels (hereinafter referred to as pixels). Figures 6-8 In the example, event camera 2 has, for example, 1280 pixels in the horizontal direction (x is 1 to 1280) and 720 pixels in the vertical direction (y is 1 to 720). Figure 3 The area 40 shown varies depending on the distance between the event camera 2 and the black-and-white subject 4, but is generally a sufficiently large area compared to the black-and-white subject 4 (QR code). Therefore, in Figure 3 The image shows only a portion of the photographed surface 40. As an example of the size of the photographed surface 40 and the size of the black-and-white subject 4 (QR code), it is easy to understand... Figure 11 The image. For example, if the size of the photographed surface 40 is 128 mm in the horizontal direction and 72 mm in the vertical direction, 0.01 mm square of the photographed surface 40 corresponds to one pixel of the event camera 2.

[0087] like Figure 5 As shown, when a change in brightness (luminance) occurs (triggered), event camera 2 records the coordinates x and y of the pixel where the change occurred, the polarity value p of the brightness (luminance) change, and the trigger time t as event data D. Therefore, if no trigger occurs, event camera 2 does not make any new recordings, and event data D is not updated. On the other hand, if a trigger occurs, the trigger is recorded in a very short time. Although this varies depending on the performance of event camera 2, it is possible, for example, to record within 1 microsecond (1 / 1,000,000th of a second).

[0088] exist Figure 5In the example, in the pixel with x-coordinate 1 and y-coordinate Y-1 (719 in the 720-pixel example), an event (trigger) was recorded where the polarity value p changed to 1 at 0.5 milliseconds (0.5 milliseconds). Additionally, in the pixel with x-coordinate X (1280 in the 1280-pixel example) and y-coordinate Y-1 (719), a trigger was also recorded where the polarity value p changed to 0 at 0.5 milliseconds. At 0.7 milliseconds, a trigger was recorded where the polarity value p changed to 1 in the pixel with x-coordinate 2 and y-coordinate Y-1 (719), and at 1 millisecond, a trigger was also recorded where the polarity value p changed to 1 in the pixel with x-coordinate 2 and y-coordinate Y-1 (719).

[0089] Furthermore, the event camera 2 detects changes in brightness (luminance), but not the absolute value of brightness (luminance). As described above, the amount of brightness change is detected logarithmically. However, for example, if the brightness (luminance) is decomposed into 128 segments from black (0) to white (1), and the threshold for detecting change is 10 / 128, a polarity value p of "1" is recorded when the brightness increases by more than 10 / 128 from its previous value, and a polarity value p of "0" is recorded when the brightness decreases by more than 10 / 128. In this example, if the absolute value of brightness (luminance) is 0.5 before the change and 0.3 after the change, a polarity value p of "0" is recorded because it exceeds the darkening threshold. However, if the absolute value of brightness (luminance) is 0 before the change and 0.3 after the change, a polarity value p of "1" is recorded because it exceeds the brightening threshold. That is, even if the absolute value of brightness (luminance) is the same and is 0.3, the recorded polarity value p is either "0" or "1".

[0090] exist Figure 5 and Figure 6 In the relationship, Figure 6 The brightness arrangement list L is shown at a time of 1 millisecond. No brightness (lightness) change (trigger) occurs in pixels other than the three pixels with x-coordinate 1 and y-coordinate Y-1 (719), x-coordinate X (1280) and y-coordinate Y-1 (719), and x-coordinate 2 and y-coordinate Y-1 (719), so the time remains "0 milliseconds". In the pixels with x-coordinate 1 and y-coordinate Y-1 (719), and x-coordinate X (1280) and y-coordinate Y-1 (719), the brightness (lightness) change (trigger) occurs at 0.5 milliseconds, so the timestamp T is set to "0.5 milliseconds", and the polarity value p is set to "1" and "0" respectively. In the pixel with x-coordinate 2 and y-coordinate Y-1 (719), the brightness (lightness) change (trigger) occurs twice, at 0.7 milliseconds and 1 millisecond, so in Figure 5 Each record contains its own timestamp T. Additionally, the state representing 1 millisecond... Figure 6 In the example, the polarity value p is recorded as "1" and the timestamp T is recorded as "1 millisecond".

[0091] Secondly, event camera 2 detects changes in brightness (luminance), so if there is no change, nothing is detected. Therefore, as long as the black and white subject 4 (QR code) remains stationary, nothing is detected. Event data D is recorded only when the black and white subject 4 (QR code) moves.

[0092] The size of the photographed surface 40 is 128 mm in the horizontal direction and 72 mm in the vertical direction. In the above hypothetical example where a 1 mm square of the photographed surface 40 corresponds to 100 pixels of the event camera 2, when the width of the white and black quadrilateral cells of the black-and-white subject 4 (QR code) is 1 mm, one cell of the black-and-white subject 4 (QR code) corresponds to 100 pixels of the event camera 2. This means that if a square is used to represent one pixel of the photographed surface 40 receiving light, then one side is 0.1 mm. Furthermore, if the black-and-white subject 4 (QR code) moves, for example, 10 cm per second, it is equivalent to moving 100 mm per second, which is equivalent to moving 1000 pixels per second. That is, the edge of a cell of the black-and-white subject 4 (QR code) moves 1000 pixels in the relative motion direction M within one second. If measured in pixels, the edge of a cell of the black-and-white subject 4 (QR code) passes through within 1 millisecond.

[0093] Furthermore, "the edge of a cell in a black and white subject 4 (QR code)" refers to the boundary between a white cell and a black cell. From the perspective of a white cell, there are boundary areas where a black cell ends and a white cell begins, as well as boundary areas where a white cell ends and a black cell begins. Similarly, from the perspective of a black cell, the beginning and end are boundary areas.

[0094] Assuming the resolution of event camera 2 is 0.1 milliseconds (0.1 milliseconds), when a pixel passes through the edge of a cell in the black and white subject 4 (QR code), it can record 10 changes in brightness (luminance). For example, when the original brightness moves from a black cell (luminance value 0) to a white cell (luminance value 1), as long as a pixel at a specific coordinate in event camera 2 is located at a position where the entire change in brightness (luminance) can be observed, that pixel can detect 10 changes in brightness (luminance) from the time when the black cell of the black and white subject 4 (QR code) is displayed in the entire area (time 0 milliseconds) to the time when the white cell of the black and white subject 4 (QR code) is displayed in the entire area (time 1 millisecond).

[0095] Therefore, in this example, the event camera 2 detects 10 changes in a single pixel, from a state where the entire pixel is black (brightness value 0) (time 0 milliseconds) to a state where the entire pixel is white (brightness value 1) (time 1 millisecond). These 10 changes (from time 0 milliseconds to time 1 millisecond) have a threshold of brightness (luminance), so the coordinates x, y, polarity value p "1", and trigger time t of that specific pixel are recorded in the event data D.

[0096] In the above-described configuration, one edge of a cell in the black-and-white subject 4 (QR code) corresponds to 10 pixels of the event camera 2. Assuming these 10 pixels are continuous in the relative motion direction M, the initial first pixel and the final tenth pixel correspond to the edge of the cell in the black-and-white subject 4 (QR code), allowing for the detection of black-and-white inversion. On the other hand, the second to ninth pixels do not correspond to the edge of the cell in the black-and-white subject 4 (QR code), thus no black-and-white inversion occurs. In this case, the initial first pixel records 10 times (from time 0 ms to time 1 ms), with coordinates x, y, polarity p "1", and trigger time t. In the second to ninth pixels, there is no brightness (luminance) change exceeding the threshold, therefore event data D is not recorded. The final tenth pixel transitions from a white (luminance value 1) cell to a black (luminance value 0) cell, thus exhibiting a brightness (luminance) change exceeding the threshold, and the coordinates x, y, polarity p "0", and trigger time t are recorded.

[0097] In other words, in the edge of the black-and-white inverted cell of the black-and-white subject 4 (QR code), a change in brightness (luminance) above the threshold is detected and a polarity value p "1" indicating a change in brightness (luminance) above the threshold in the bright direction and a polarity value p "0" indicating a change in brightness (luminance) above the threshold in the dark direction are recorded. However, in the middle part of the continuous white or black cell, there is no change in brightness (luminance), so it is not recorded.

[0098] Furthermore, the above explanation assumes that only one cell in a monochrome subject 4 (QR code) changes from white to black. However, in actual monochrome subject 4 (QR codes), multiple white or black cells are usually consecutive. Therefore, the number of pixels at the edges of cells in a monochrome subject 4 (QR code) is actually less.

[0099] This is crucial for detecting a black and white subject 4 (QR code) from the photographed surface 40. In a black and white subject 4 (QR code), the cells are either white or black with no intermediate colors. However, in other areas, such as parts printed with the black and white subject 4 (QR code), the brightness (luminance) varies randomly. Therefore, it is rare to maintain white (luminance value 1) or black (luminance value 0) for even a few milliseconds.

[0100] Based on the above technical considerations, the image system 1 of this disclosure will be described. For example... Figure 1 , Figure 2 As shown, event camera 2 captures a black-and-white subject 4 (QR code) as the subject. The black-and-white subject 4 (QR code) moves at a speed of, for example, 10 centimeters per second in a constant relative direction of motion M. Of course, a speed of 1-10 centimeters per second is just an example; much faster movements are possible. More important than the physical speed of the black-and-white subject 4 (QR code) is the change in the field of view observed by event camera 2. If the distance between event camera 2 and the black-and-white subject 4 is close, even faster movements can be detected.

[0101] When the event camera 2 detects a change in brightness (luminance), it outputs event data D corresponding to that change to the image restoration device 3. The event data D records, for example... Figure 5 The graph shows the coordinates x and y of the pixel where the event occurred, the polarity value p(0, 1) of the brightness (luminance) change that exceeds the threshold, and the trigger time t (timestamp T) of the event.

[0102] As mentioned above, Figure 1 The hardware of the image restoration device 3 is shown, with the memory 10 and processor 12 connected via a bus. Figure 2 The software function blocks of the image restoration device 3 are shown, including an initialization block 100, an update block 120, a correction block 140, and an output block 160. The aforementioned event data D is recorded in the memory 10. Figure 6 It is a record Figure 5 The example of event data D shown records the pixel coordinates x, y of the event camera 2, the polarity value p(0,1) of the brightness (luminance) change that occurs above the threshold, and the trigger time t (timestamp T) of the event in the brightness sorting list L of memory 10.

[0103] Initialization block 100 initializes the data in memory 10. Initialization means setting the timestamp T to 0 in the x, y coordinates of all pixels in event camera 2. In the case of an event camera 2 resolution of, for example, 0.1 milliseconds, 0 represents 0 milliseconds. Additionally, initialization means setting the brightness value I to an intermediate value m that is neither "0" nor "1" in the x, y coordinates of all pixels in event camera 2. In this example, "0.5" is used as the intermediate value m. An example of the brightness arrangement list L of the initialized memory 10 is as follows: Figure 7 In the x, y coordinates of all pixels of event camera 2, the brightness value I is "0.5" and the timestamp T is "0". The actions performed through this initialization block 100 are related to... Figure 12 The flowchart corresponds to step S101.

[0104] Update block 120 retrieves event data D from event camera 2 and overwrites (updates) the data in memory 10. For example, when retrieving... Figure 5 When event data D is received, the brightness sorting list L of memory 10 is in a state with a time interval of 0.5 milliseconds, as follows: Figure 8 Update as shown, with a time interval of 1 millisecond. Figure 6 The update is shown below. When focusing on the coordinates with x-coordinate of 2 and y-coordinate of Y-1, the state at time 0.5 milliseconds ( Figure 8 In the current state, no trigger has occurred, the brightness value I is "0.5", and the timestamp T is "0 milliseconds".

[0105] In addition, Figure 5 , Figure 6 , Figure 8 In the event, an event occurs at the end of the brightness arrangement list L ((x=1, y=Y-1), (x=2, y=Y-1), (x=X, y=Y-1)), but this is caused by the black and white subject 4 moving from the end of the field of view of the event camera 2. Furthermore, if the black and white subject 4 is a QR code, the QR code needs to enter the field of view of the event camera 2 by a specified amount in order to read the code.

[0106] exist Figure 9 In the example, the trigger coordinates xf and yf are the pixels represented in the image as black (brightness value I: "0") or white (brightness value I: "1"). Furthermore, the action performed through this update block 120 is related to... Figure 12 The flowchart corresponds to step S102.

[0107] To quickly extract the black and white subject 4 (QR code) from the event data D, correction block 140 performs a second sub-correction process in addition to the first sub-correction process as part of the main process, for a total of two corrections to the brightness arrangement list L of memory 10. As described above, the black and white subject 4 (QR code) only includes white and black cells, with no intermediate colors. Therefore, the pixels of event camera 2 detect black and white inversion at the edges of the cells, but not at the edges. Furthermore, the cells of the black and white subject 4 (QR code) are printed as straight lines, so the trigger coordinates xf and yf of the edges of the cells of the black and white subject 4 (QR code) are either white or black straight lines. Figure 9 In the example, the black line and the white line are the trigger coordinates xf and yf. Here, in the first sub-correction process, the white line and the black line are detected by trigger coordinates xf and yf.

[0108] In addition, Figure 9The area to the lower left of the black vertical line and the lower black horizontal line within the trigger coordinates xf, yf is white. However, this white area only represents the image of the moving direction of the black and white subject 4 (QR code) and may not actually be the trigger coordinates xf, yf. The shooting time varies depending on the number of pixels, the size of the cells of the black and white subject 4 (QR code), the relative movement direction M, and the relative movement speed, but is set within a specified allowable time range. Figure 9 In the example, the widths of the white and black lines within the black and white subject 4 (QR code) correspond to the specified allowable time range.

[0109] In addition, Figure 9 In the diagram, besides the white portion mentioned above, the areas other than the black and white subject 4 (QR code) are designated as non-trigger coordinates xn, yn, where no triggering occurs. The intermediate coordinates xm, ym, with a brightness value I of "0.5", are used. However, in reality, not only the black and white subject 4 (QR code) moves relatively, but the components printed with the black and white subject 4 (QR code) also move relatively. Therefore, even areas other than the black and white subject 4 (QR code) are usually designated as trigger coordinates xf, yf in relation to the movement of objects such as components. However, the brightness (luminance) changes in areas other than the black and white subject 4 (QR code) are random, and pixels detecting changes in brightness (luminance) in either the light or dark direction are usually not configured as linear lines. Furthermore, there are no changes in brightness (luminance) in areas where the object does not move, so the intermediate coordinates xm, ym, with a brightness value I of "0.5", remain unchanged.

[0110] Through the first sub-correction process described above, the edges of the cells of the black and white subject 4 (QR code) can be determined. Furthermore, within the area containing the black and white subject 4 (QR code), the non-trigger coordinates xn and yn, where the intermediate coordinates xm and ym have a brightness value I of "0.5", are areas where the black and white inversion has not occurred; these are either white or black in the actual QR code cells. The second sub-correction process in this example is to set the intermediate coordinates xm and ym with a brightness value I of "0.5" as trigger coordinates xf and yf, where the brightness value I changes from "1" to "0.5" or from "0" to "0.5" in the relative motion direction M, as the focus coordinates xc and yc, and the trigger coordinates xf and yf, where the brightness value I becomes the same in the opposite direction O from the focus coordinates xc and yc.

[0111] For example, starting from the coordinates xc, yc of the continuous focus coordinates that trigger a change in brightness (luminance) to black in the dark direction, the same brightness value I of "0" is replaced in the opposite direction O to the relative motion direction M. Figure 10 In the example, starting from the left-hand focus coordinates xc, yc in the opposite direction O to the relative motion direction M, the gray color is replaced with black. Conversely, starting from the right-hand focus coordinates xc, yc in the opposite direction O to the relative motion direction M, the gray color is replaced with white. Therefore, the QR code cell output will be either white or black.

[0112] Figure 11 The image shows a black and white subject 4 (QR code) undergoing the second sub-correction process. As described above, the first sub-correction process, which is the main process, is the process of detecting the edges of the QR code cells; therefore, this process is usually not performed in locations where the black and white subject 4 (QR code) is not present. Furthermore, the second sub-correction process is performed only in the areas determined by the first sub-correction process to be black and white subject 4 (QR code); therefore, this second sub-correction process is also not performed in locations where the black and white subject 4 (QR code) is not present. Therefore, in Figure 11 In the areas other than the black and white subject 4 (QR code), where the object moves, the changes in brightness (luminance) are preserved as is, and white pixels (brightness value I is "1") and black pixels (brightness value I is "0") exist randomly.

[0113] However, in order to more accurately capture the black and white subject 4 (QR code) within the field of view of event camera 2, such as Figure 11 As shown, a process is performed to remove the randomness and display it in white. Similarly, in the non-moving parts of objects such as parts, the following process is performed: that is, the intermediate coordinates xm, ym with a brightness value I of "0.5" are retained, but the intermediate coordinates xm, ym of such parts are also displayed in white. Furthermore, in Figure 11 The line extending from the black and white subject 4 (QR code) to the lower left simply represents the relative motion direction M of the black and white subject 4 (QR code), and is not a line obtained through the second sub-correction process, nor does it represent the shooting time.

[0114] The first and second sub-correction processes described above are performed using correction block 140. The actions within correction block 140 are... Figure 12 Step S103 of the flowchart.

[0115] Output block 160 output as follows Figure 11 The image shows the correction process. As described above, white and black cells are displayed in areas where the black and white subject 4 (QR code) exists. In areas where the black and white subject 4 (QR code) does not exist, white (brightness value I is "1") pixels and black (brightness value I is "0") pixels are randomly present, or areas without change are set to white. In either case, the intermediate coordinates xm and ym with a brightness value I of "0.5" do not exist, therefore binary image B is output. Figure 12 In the flowchart, the action in output block 160 corresponds to step S104. Furthermore, in Figure 11 In the example, a process was performed to correct the parts of the black and white subject 4 (QR code) to white. However, as long as the color is easy to control, it can be black, green or other colors.

[0116] Next, the information contained in the black and white subject 4 (QR code) is read from the output binary image B. The black and white subject 4 (QR code) has a fine pattern 4a in three locations, so it is easy to detect the fine pattern 4a of a special shape from randomly existing white (brightness value I is "1") and black (brightness value I is "0") pixels in software. Moreover, when the position of the fine pattern 4a is known, it is easy to calculate whether the brightness at the center position of the cell of the black and white subject 4 (QR code) is white (brightness value I is "1") or black (brightness value I is "0"), and the information of the black and white subject 4 (QR code) can be read.

[0117] In particular, in this disclosure, a first sub-correction process is used to determine the region where the black and white subject 4 (QR code) exists, and a second sub-correction process is performed in this region. Therefore, the black and white subject 4 (QR code) can be repaired before actually reading the entire black and white subject 4 (QR code) (after reading the minimum two pixels). Furthermore, the repaired information is a binary image B, so there is no need to perform a binarization process again. Therefore, according to this disclosure, information reading of the black and white subject 4 (QR code) can be performed within a very short shooting time (minimum 1 microsecond).

[0118] However, if mastering the black and white subject 4 (QR code) takes a certain amount of time, the second sub-correction process can be omitted, and the cells of the black and white subject 4 (QR code) can be detected solely through the first sub-correction process. That is, in Figure 9 and Figure 10 In the example, the thickness of the line corresponds to the allowable time range, but by lengthening that allowable time range, an entire cell of the black and white subject 4 (QR code) can be made white or black. In this sense, the first sub-correction process is the main process.

[0119] (Second Implementation)

[0120] like Figure 13 As shown, the second embodiment is a variation of the first embodiment.

[0121] The image restoration process of the second embodiment executes S202 instead of S102. In this S202, update block 120 skips a portion of the update process of the luminance arrangement list L corresponding to the event data D of each event until multiple consecutive events with the same trigger coordinates xf, yf, and polarity value p are triggered. The skipped portion of the update process refers to the overwrite operation using the polarity value p and the trigger time t. Therefore, in the continuous triggering of events with the same trigger coordinates xf, yf, and polarity value p, update block 120 overwrites the luminance value I of the trigger coordinates xf, yf in the luminance arrangement list L with the same polarity value p. At the same time, in this continuous triggering, update block 120 overwrites the timestamp T of the trigger coordinates xf, yf in the luminance arrangement list L with the trigger time t of the last event in the continuous triggering.

[0122] In this S202, the number of consecutive events in which the update block 120 skips the update process is set, for example, to an appropriate number that can suppress overwriting errors of the polarity value p caused by noise triggering such as external disturbances. Furthermore, apart from the overwriting operation of the polarity value p and the triggering time t, the update process of the update block 120 in S202 is performed in the same manner as in the first embodiment. In this second embodiment, S202 corresponds to the "update process".

[0123] (Effects)

[0124] The effects of the second embodiment described above will be explained below.

[0125] According to the second embodiment, the update of the brightness arrangement list L is skipped until the event with the same coordinates xf, yf, and polarity value p is triggered multiple times consecutively. This suppresses poor restoration of the binary image B caused by overwriting errors of the polarity value p due to noise triggering.

[0126] The second embodiment will be described again from another perspective. In the first embodiment, the premise was that the changes in brightness (luminance) at the edges of the black-and-white subject 4 (QR code) were primarily captured, while the areas outside the edges remained continuously white or black without any changes in brightness (luminance). This premise is certainly correct, but the possibility of noise caused by external disturbances cannot be ruled out.

[0127] For example, in the white portion of a cell in a black-and-white subject 4 (QR code), the possibility cannot be ruled out that a pixel might detect a decrease in brightness (luminance) above a threshold due to changes in external light or dirt. Furthermore, even without changes in light, there is a possibility that the analog output signal from the constant voltage of the event camera 2 might carry noise from the outside. In this case, the pixel that detects a decrease in brightness (luminance) or noise, along with its coordinates x, y and polarity value p "0", is recorded along with the trigger time t (timestamp T) in the event data D. As a result, in the brightness arrangement list L of memory 10, the coordinates of this pixel are the trigger coordinates xf, yf, and the brightness value I is also "0".

[0128] However, since the pixel coordinates do not change in brightness (luminance), the white portion of the cell in the black and white subject 4 (QR code) should remain unchanged and be at non-trigger coordinates xn, yn. The brightness value I should be at the intermediate coordinates xm, ym of the intermediate value m representing "0.5". Therefore, when erroneous information is recorded in the brightness arrangement list L of memory 10, a filtering process to remove noise through the first and second correction steps in correction block 140 is required, which prolongs the processing time or may lead to misjudgment.

[0129] Even so, the likelihood of detecting this noise multiple times consecutively within the same pixel is low. This is because the white portion of the cell in the black and white subject 4 (QR code) should remain unchanged, rather than a region where the brightness (luminance) continuously decreases. On the other hand, at the edges of the cells in the black and white subject 4 (QR code), the defined pixels are capable of detecting multiple changes in brightness (luminance).

[0130] Therefore, a pixel system that detects multiple consecutive changes in brightness (luminance) is more likely to detect the edges of cells in a black and white subject 4 (QR code). On the other hand, a pixel system that detects a one-off change in brightness (luminance) is more likely to erroneously detect a brightness change that did not actually occur or be affected by external noise.

[0131] The second implementation utilizes this characteristic. Even if a change in brightness (luminance) occurs at a specific pixel (specific coordinates), as long as the polarity values ​​p in the same direction ("0" or "1") are not consecutively updated multiple times, the brightness arrangement list L in the memory 10 of update block 120 will not be updated. This is Figure 13 Step S202 of the flowchart.

[0132] Thus, the rewriting of the brightness value I to "0" or "1" in the brightness arrangement list L of memory 10 is the result of multiple consecutive events, therefore the reliability of the brightness arrangement list L is high. Therefore, the correction block 140 can use high-precision information to perform the first sub-correction process and the second sub-correction process, and can eliminate or reduce the filtering process.

[0133] As a result, processing time can be reduced and read errors can be decreased.

[0134] Furthermore, the number of consecutive iterations can be appropriately determined based on factors such as the pixel count of the event camera 2, the area of ​​the photographed surface 40, the cell size of the black-and-white subject 4 (QR code), and the moving speed of the black-and-white subject 4 (QR code) in the relative motion direction M. For example, the number of consecutive iterations is preferably one that can sufficiently detect brightness changes based on the edges of the black-and-white subject 4 (QR code).

[0135] The above describes several embodiments, but this disclosure is not limited to these embodiments and can be interpreted in various ways and combinations without departing from the spirit of this disclosure.

[0136] The modified image restoration apparatus 3 can also be a dedicated computer configured as a processor, including at least one of digital circuitry and analog circuitry. Here, digital circuitry specifically refers to at least one of, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Furthermore, such digital circuitry may also include a memory storing programs.

[0137] In a variation, the update steps of update block 120 and S102 are not limited to the time difference between the trigger time t and the timestamp T; the brightness values ​​I of the non-trigger coordinates xn and yn in the brightness arrangement list L can also be maintained. In a variation, the second sub-correction steps of correction block 140 and S103 can also be skipped or always omitted as needed.

[0138] In a modified example, the direction of the light flow that occurs during shooting due to the relative motion with respect to the event camera 2 may also differ in a portion of the subject surface 40 captured by the event camera 2. In this case, errors in the brightness value I caused by the different direction of a portion of the light flow can also be corrected for the brightness arrangement list L, for example, according to the relative motion direction M of the black and white subject 4 with respect to the event camera 2, in the correction block 140 and the second sub-correction process of S103.

[0139] In a variation, the object to be repaired for the binary image B may be a black-and-white subject 4 other than a planar QR code that moves relative to the event camera 2 during the shooting period. Specifically, for example, the binary image B may be repaired based on event data D output by the event camera 2, which captures black-and-white subjects 4 existing around a moving body such as a vehicle.

[0140] Explanation of reference numerals in the attached figures

[0141] 2: Event Camera

[0142] 3: Image restoration device

[0143] 4: Black and white subjects

[0144] 10: Memory

[0145] 12: Processor

[0146] 40: The side being photographed

[0147] 100: Initialization block

[0148] 120: Update block

[0149] 140: Correction Block

[0150] 160: Output Block

[0151] B: Binary image

[0152] D: Event Data

[0153] I: Brightness value

[0154] L: Brightness Ranking List

[0155] S: Search direction

[0156] T: Timestamp

[0157] m: median value

[0158] p: Polarity

[0159] t: Triggering time

[0160] x, y: pixel coordinates

[0161] xc, yc: Focus on coordinates

[0162] xf, yf: Trigger coordinates

[0163] xm, ym: intermediate coordinates

[0164] xn, yn: Non-trigger coordinates

Claims

1. An image restoration apparatus for restoring a binary image (B) based on the pixel coordinates (x, y) and polarity value (p) of an event, which is output from an event camera (2) that captures a black-and-white subject (4) in relative motion during the shooting period, in association with the triggering time (t) of an event in which a brightness change occurs in at least one camera pixel. The black-and-white subject is a QR code or barcode, and it has multiple white cells representing light colors and multiple black cells representing dark colors. At the boundaries between the white and black cells, the black and white colors are reversed. The brightness of the regions in the binary image where the black-and-white subject is not present varies randomly. The image restoration device (3) comprises: The initialization unit (100) initializes the brightness value of each pixel coordinate to the intermediate value in a brightness sorting list (L) that stores any one of the polarity values ​​of the brightness increase direction corresponding to the white brightness value of the binary image, the polarity value of the brightness decrease direction corresponding to the black brightness value of the binary image, and the intermediate value between the polarity values ​​of the brightness increase direction and the polarity values ​​of the brightness decrease direction. The update unit (120) updates the brightness arrangement list initialized by the initialization unit according to the pixel coordinates and polarity values ​​of each event; The correction unit (140) detects the boundary between the white cell and the black cell using the brightness arrangement list updated by the update unit throughout the shooting period; as well as The output unit (160) outputs the brightness arrangement list updated by the update unit throughout the entire shooting period as the binary image output. In the update of the update unit, the polarity value of the event is used to overwrite the brightness value of the trigger coordinate of the pixel coordinate that is the trigger coordinate of the event in the brightness arrangement list, while the brightness value of the non-trigger coordinate of the pixel coordinate other than the trigger coordinate is maintained in the brightness arrangement list, thereby detecting the black and white subject by means of the boundary between the white cell and the black cell.

2. The image restoration apparatus according to claim 1, wherein, Skip the update of the update section until the event with the same trigger coordinate and polarity value is triggered multiple times consecutively.

3. The image restoration apparatus according to claim 1, wherein, The binary image of the black-and-white subject, whose direction of optical flow during the shooting period is the same as that of the subject surface (40) captured by the event camera through relative motion with respect to the event camera, is corrected.

4. The image restoration apparatus according to claim 1, wherein, The binary image is repaired by using a QR code that moves relative to the event camera during the shooting period as the black-and-white subject.

5. An image restoration apparatus that restores a binary image (B) based on the pixel coordinates (x, y) and polarity value (p) of an event that occurs in at least one camera pixel and is output in association with the trigger time (t) of an event in which a brightness change occurs in a black-and-white subject (4) undergoing relative motion during the shooting period from an event camera (2), wherein the polarity value is used to define the direction of increase or decrease of brightness change above a threshold by means of binary values. The image restoration apparatus (3) comprises: The initialization unit (100) initializes the brightness value of each pixel coordinate to the intermediate value in a brightness sorting list (L) that stores any one of the polarity values ​​and their intermediate values ​​as the brightness value of each pixel coordinate. The update unit (120) updates the brightness arrangement list initialized by the initialization unit according to the pixel coordinates and polarity values ​​of each event; as well as The output unit (160) outputs the brightness arrangement list updated by the update unit throughout the entire shooting period as the binary image output. In the update process of the update unit, the brightness value of the trigger coordinate of the pixel coordinate that is the trigger coordinate of the event in the brightness arrangement list is overwritten using the polarity value of the event, while the brightness value of the non-trigger coordinate of the pixel coordinate that is the trigger coordinate in the brightness arrangement list is maintained. The image restoration apparatus further includes a correction unit (140), which corrects the brightness value of the intermediate coordinate of the pixel coordinate that stores the intermediate value in the brightness arrangement list updated by the update unit throughout the entire shooting period. In the correction process of the correction unit, the relative motion direction (M) of the black and white subject relative to the event camera is set as the search direction (S). The unit searches for the coordinate of interest that is adjacent to the intermediate coordinate in the search direction among the pixel coordinates that have any polarity value stored in the brightness arrangement list. The polarity value of the coordinate of interest is overwritten and calculated onto the brightness value of the intermediate coordinate in the brightness arrangement list.

6. The image restoration apparatus according to claim 5, wherein, In the update of the update unit, the brightness value of the non-trigger coordinate switched from the trigger coordinate in the brightness arrangement list is maintained within an allowed time range from the trigger time of the coordinate before the switch, while outside the allowed time range from the trigger time of the coordinate before the switch, the intermediate value is used for overwrite operation.

7. The image restoration apparatus according to any one of claims 5 to 6, wherein, Skip the update of the update section until the event with the same trigger coordinate and polarity value is triggered multiple times consecutively.

8. The image restoration apparatus according to claim 5, wherein, The binary image of the black-and-white subject, whose direction of optical flow during the shooting period is the same as that of the subject surface (40) captured by the event camera through relative motion with respect to the event camera, is corrected.

9. The image restoration apparatus according to claim 5, wherein, The binary image is repaired by using a QR code that moves relative to the event camera during the shooting period as the black-and-white subject.

10. An image restoration method, executed by a processor (12), for restoring a binary image (B) based on the pixel coordinates (x, y) and polarity value (p) of an event, which is output from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, in association with the triggering time (t) of an event in which a brightness change occurs in at least one camera pixel. The black-and-white subject is a QR code or barcode, and it has multiple white cells representing light colors and multiple black cells representing dark colors. At the boundaries between the white and black cells, the black and white colors are reversed. The brightness of the regions in the binary image where the black-and-white subject is not present varies randomly. The image restoration method comprises: In the initialization process (S101), the brightness value of each pixel coordinate is initialized to the intermediate value in a brightness sorting list (L) that stores any one of the polarity values ​​of the brightness increase direction corresponding to the white brightness value of the binary image, the polarity value of the brightness decrease direction corresponding to the black brightness value of the binary image, and the intermediate value between the polarity values ​​of the brightness increase direction and the polarity values ​​of the brightness decrease direction. The update process (S102, S202) updates the brightness arrangement list initialized by the initialization process according to the pixel coordinates and polarity values ​​of each event; In the correction process (S103), the boundary between the white cell and the black cell is detected using the brightness arrangement list updated throughout the shooting process by the update process. as well as The output process (S104) outputs the brightness arrangement list updated throughout the entire shooting period as the binary image output via the update process. In the update process, the polarity value of the event is used to overwrite the brightness value of the trigger coordinate of the pixel coordinate that is the trigger coordinate of the event in the brightness arrangement list, while the brightness value of the non-trigger coordinate of the pixel coordinate other than the trigger coordinate is maintained in the brightness arrangement list, thereby detecting the black and white subject by utilizing the boundary between the white cell and the black cell.

11. The image restoration method according to claim 10, wherein, Skip the update process (S202) until the event with the same trigger coordinate and polarity value is triggered multiple times consecutively.

12. The image restoration method according to claim 10, wherein, The image restoration method is executed by the processor to restore the binary image of the black-and-white subject whose direction of optical flow during the shooting period is the same in the entire area of ​​the photographed surface (40) captured by the event camera, due to the relative motion of the event camera.

13. The image restoration method according to claim 10, wherein, The image restoration method is executed by the processor to restore the binary image by using a QR code that moves relative to the event camera during the shooting period as the black-and-white subject.

14. An image restoration method, executed by a processor (12), restores a binary image (B) based on the pixel coordinates (x, y) and polarity value (p) of an event, output in association with the trigger time (t) of an event in which a brightness change occurs in at least one camera pixel, from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, wherein the polarity value, by means of binary, specifies the direction of increase or decrease of the brightness change above a threshold, and comprises: In the initialization process (S101), the brightness value of each pixel coordinate is initialized to the intermediate value in a brightness sorting list (L) where any one of the polarity values ​​and their intermediate values ​​is stored as the brightness value of each pixel coordinate. The update process (S102, S202) updates the brightness arrangement list initialized by the initialization process according to the pixel coordinates and polarity values ​​of each event; as well as The output process (S104) outputs the brightness arrangement list updated throughout the entire shooting period as the binary image output via the update process. In the update process, the polarity value of the event is used to overwrite the brightness value of the trigger coordinate of the pixel coordinate that is the trigger coordinate of the event in the brightness arrangement list, while the brightness value of the non-trigger coordinate of the pixel coordinate that is the trigger coordinate is retained in the brightness arrangement list. The image restoration method further includes a correction step (S103), in which the brightness value of the intermediate coordinate of the pixel coordinates storing the intermediate value is corrected in the brightness arrangement list updated by the update step throughout the entire shooting period. In the correction process, the relative motion direction (M) of the black and white subject relative to the event camera is set as the search direction (S). The search is performed on the pixel coordinates that are adjacent to the intermediate coordinate in the search direction among the pixel coordinates that have any polarity value stored in the brightness arrangement list. The polarity value of the focus coordinate is overwritten and calculated onto the brightness value of the intermediate coordinate in the brightness arrangement list.

15. The image restoration method according to claim 14, wherein, In the update process, the brightness value of the non-trigger coordinate switched from the trigger coordinate in the brightness arrangement list is maintained within an allowed time range from the trigger time of the coordinate before the switch, while outside the allowed time range from the trigger time of the coordinate before the switch, the intermediate value is used for overwriting.

16. The image restoration method according to any one of claims 14 to 15, wherein, Skip the update process (S202) until the event with the same trigger coordinate and polarity value is triggered multiple times consecutively.

17. The image restoration method according to claim 14, wherein, The image restoration method is executed by the processor to restore the binary image of the black-and-white subject whose direction of optical flow during the shooting period is the same in the entire area of ​​the photographed surface (40) captured by the event camera, due to the relative motion of the event camera.

18. The image restoration method according to claim 14, wherein, The image restoration method is executed by the processor to restore the binary image by using a QR code that moves relative to the event camera during the shooting period as the black-and-white subject.

19. A storage medium storing an image restoration program, the image restoration program comprising commands executed by a processor (12) for restoring a binary image (B) based on pixel coordinates (x, y) and polarity values ​​(p) of an event, output in association with a trigger time (t) of an event in which a brightness change occurs in at least one camera pixel, from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, wherein... The black-and-white subject is a QR code or barcode, and it has multiple white cells representing light colors and multiple black cells representing dark colors. At the boundaries between the white and black cells, the black and white colors are reversed. The brightness of the regions in the binary image where the black-and-white subject is not present varies randomly. The command includes: In the initialization process (S101), the brightness value of each pixel coordinate is initialized to the intermediate value in a brightness sorting list (L) that stores any one of the polarity values ​​of the brightness increase direction corresponding to the white brightness value of the binary image, the polarity value of the brightness decrease direction corresponding to the black brightness value of the binary image, and the intermediate value between the polarity values ​​of the brightness increase direction and the polarity values ​​of the brightness decrease direction. The update process (S102, S202) updates the brightness arrangement list initialized by the initialization process according to the pixel coordinates and polarity values ​​of each event; In the correction process (S103), the boundary between the white cell and the black cell is detected using the brightness arrangement list updated throughout the shooting process by the update process. as well as The output process (S104) outputs the brightness arrangement list updated throughout the entire shooting period as the binary image output via the update process. In the update process, the polarity value of the event is used to overwrite the brightness value of the trigger coordinate of the pixel coordinate that is the trigger coordinate of the event in the brightness arrangement list, while the brightness value of the non-trigger coordinate of the pixel coordinate other than the trigger coordinate is maintained in the brightness arrangement list, thereby detecting the black and white subject by utilizing the boundary between the white cell and the black cell.

20. The storage medium according to claim 19, wherein, Skip the update process (S202) until the event with the same trigger coordinate and polarity value is triggered multiple times consecutively.

21. The storage medium according to claim 19, wherein, The image restoration procedure is used to restore the binary image of the black-and-white subject whose direction of optical flow occurs during the shooting period by relative motion with respect to the event camera is the same throughout the entire area of ​​the photographed surface (40) captured by the event camera.

22. The storage medium according to claim 19, wherein, The image restoration procedure is used to restore the binary image by taking a QR code that moves relative to the event camera during the shooting period as the black-and-white subject.

23. A storage medium storing an image restoration program, the image restoration program comprising commands executed by a processor (12) for restoring a binary image (B) based on pixel coordinates (x, y) and polarity values ​​(p) of an event, output in association with a trigger time (t) of an event in which a brightness change occurs in at least one camera pixel, from an event camera (2) capturing a black-and-white subject (4) in relative motion during the shooting period, wherein the polarity values, by means of binary representation, specify the direction of increase or decrease of the brightness change above a threshold, wherein... The command includes: In the initialization process (S101), the brightness value of each pixel coordinate is initialized to the intermediate value in a brightness sorting list (L) where any one of the polarity values ​​and their intermediate values ​​is stored as the brightness value of each pixel coordinate. The update process (S102, S202) updates the brightness arrangement list initialized by the initialization process according to the pixel coordinates and polarity values ​​of each event; as well as The output process (S104) outputs the brightness arrangement list updated throughout the entire shooting period as the binary image output via the update process. In the update process, the polarity value of the event is used to overwrite the brightness value of the trigger coordinate of the pixel coordinate that is the trigger coordinate of the event in the brightness arrangement list, while the brightness value of the non-trigger coordinate of the pixel coordinate that is the trigger coordinate is retained in the brightness arrangement list. The command further includes a correction step (S103), in which the brightness value of the intermediate coordinate of the pixel coordinate storing the intermediate value is corrected in the brightness arrangement list updated by the update step throughout the entire shooting period. In the correction process, the relative motion direction (M) of the black and white subject relative to the event camera is set as the search direction (S). The search is performed on the pixel coordinates that are adjacent to the intermediate coordinate in the search direction among the pixel coordinates that have any polarity value stored in the brightness arrangement list. The polarity value of the focus coordinate is overwritten and calculated onto the brightness value of the intermediate coordinate in the brightness arrangement list.

24. The storage medium according to claim 23, wherein, In the update process, the brightness value of the non-trigger coordinate switched from the trigger coordinate in the brightness arrangement list is maintained within an allowed time range from the trigger time of the coordinate before the switch, while the intermediate value is used for overwriting operations outside the allowed time range from the trigger time of the coordinate before the switch.

25. The storage medium according to any one of claims 23 to 24, wherein, Skip the update process (S202) until the event with the same trigger coordinate and polarity value is triggered multiple times consecutively.

26. The storage medium according to claim 23, wherein, The image restoration procedure is used to restore the binary image of the black-and-white subject whose direction of optical flow occurs during the shooting period by relative motion with respect to the event camera is the same throughout the entire area of ​​the photographed surface (40) captured by the event camera.

27. The storage medium according to claim 23, wherein, The image restoration procedure is used to restore the binary image by taking a QR code that moves relative to the event camera during the shooting period as the black-and-white subject.

Images