Information processing device, information processing method, and program

Abstract

To improve visibility in the case of reversely reproducing an address event signal.SOLUTION: An information processing device comprises: acquisition means which acquires an address event signal indicative of the address of a pixel in which luminance variation occurs and a time; and generation means which, on the basis of the address event signal, generates an image of a time sequence indicative of the position of at least one pixel in which luminance variation has occurred and the direction of the luminance variation. In the case of reversing the time sequence and displaying the image, the generation means generates an image where the direction of the luminance variation is reversed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the display of event-based sensors. [Background technology]

[0002] An event-based sensor is known that outputs a change in luminance of each pixel as an address event signal in real time (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-134271 A Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to improve visibility when an address event signal is played back in reverse. [Means for solving the problem]

[0005] The information processing device according to the present invention includes a generating means for generating a time-series image indicating a position of at least one pixel where a luminance change has occurred and a direction of the luminance change based on the address event signal, When the image is displayed in reverse chronological order, the generating means generates an image in which the direction of luminance change is reversed. Effect of the Invention

[0006] When the address event signal is played back in reverse, the visibility can be improved. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 3] A diagram showing an example of the configuration of an event-based sensor [Figure 4] A diagram showing an example of a display [Diagram 5] A diagram showing an example of a display [Figure 6] A diagram showing an example of a display [Figure 7] A diagram showing an example of a display [Figure 8] A diagram showing an example of a display [Figure 9] A diagram showing an example of a display [Figure 10] A flowchart for explaining a process executed by an information processing device. [Figure 11] A diagram showing an example of a user interface DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an information processing device according to an embodiment of the present invention will be described with reference to the drawings. In this regard, components having the same functions in all the drawings will be given the same numbers, and repeated description will be omitted.

[0009] <Embodiment 1> This article describes how to display an image generated from an address event signal that indicates the address and time of a pixel where a luminance change occurred in an information processing device using an event-based sensor. In particular, this article describes a method of displaying an image using a three-dimensional coordinate space that is composed of the coordinates (XY) of the pixel where the event occurred and an axis (T) that indicates the time (T) where the luminance change occurred (hereinafter referred to as XYT display).

[0010] <Information processing device> FIG. 1 shows a block diagram illustrating an example of a hardware configuration of an information processing device. In FIG. 1, the information processing device 100 includes an image forming optical system 1010, an event-based sensor 101 including a photoelectric conversion element 1011, a CPU 1011, and a memory 103. The event-based sensor 101 outputs an address event signal according to the received incident light. The image forming optical system 1010 is specifically a light receiving lens that receives incident light and forms an image on the photoelectric conversion element 1011. The photoelectric conversion element 1011 is specifically a SPAD sensor or a CMOS sensor that outputs an address event signal according to the received incident light. The CPU 1011 reads out and executes an OS and other programs stored in the memory 103, controls each connected component, and performs calculations and logical judgments for various processes. The processes executed by the CPU 1011 include information processing according to this embodiment. The CPU 1011 also controls the driving of the focus and iris of the image forming optical system 1010, the driving of the photoelectric conversion element 1011, and the like. The memory 103 is, for example, a hard disk drive or an external storage device, and stores programs and various data related to the information processing of the embodiment. The display unit 104 performs XYT display based on various information and address event signals.

[0011] Next, an example of the functional configuration of the information processing device according to the present embodiment will be described with reference to FIG. 2. The information processing device 100 includes an event-based sensor 101, an acquisition unit 201, an image generation unit 202, a display control unit 203, a display unit 104, and a user interface 105. The event-based sensor 101 outputs an address event signal according to the received incident light. More specifically, the event-based sensor 101 outputs an address event signal indicating the address and time of a pixel where a change in luminance has occurred. The acquisition unit 201 acquires the address event signal output by the event-based sensor. The address event signal includes information on the position of a pixel (or a group of pixels) where a change in luminance has occurred in a predetermined time range, the direction of the change in luminance, and the time when the change in luminance (event) occurred. The image generation unit 202 generates a time-series image indicating the position of at least one or more pixels where a change in luminance has occurred in a predetermined time range and the direction of the change in luminance based on the acquired address event signal. That is, the image generation unit 202 generates an XYT image from the address event signal. When the display control unit 203 displays an image in a reversed time series, the display control unit 203 displays the image in a reversed direction of the luminance change. When the display unit 104 displays the image in a reversed time series, the display unit 104 displays the image in a reversed direction of the luminance change. For example, the display unit 104 may be a device outside the information processing device, and may be, for example, a display device such as a display or a projection device such as a projector. The user interface 105 accepts various instructions such as imaging and display input by a user. Note that the display unit 104 and the operation unit 105 may be configured by an external device different from the information processing device. In addition, the functional configuration of the information processing device may include configurations other than those listed here.

[0012] <Event-based sensors> An example of the event-based sensor according to the present embodiment will be described. The event-based sensor counts the number of incident photons and determines the timing when the number of counted photons exceeds a predetermined threshold. The event-based sensor also measures the time (clock count) required for the number of photons to reach or exceed a first threshold, and detects a change in luminance by comparing the required times. Specifically, when the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is equal to or greater than a second threshold, a change in luminance in the negative direction is detected. If the difference T0-T is equal to or greater than the second threshold, a change in luminance in the positive direction is detected. If the difference between T and T0 is less than the second threshold, no change in luminance is detected. Note that the second threshold is a value equal to or greater than zero, and is set in accordance with a preset value or other parameters. A detailed configuration will be described below. FIG. 3(a) is a diagram showing an example of the configuration of the photoelectric conversion element 1011. The photoelectric conversion element 1011 is composed of a pixel unit 110 and a peripheral circuit 120. The peripheral circuit 120 includes a vertical arbitration circuit 121 and a horizontal read circuit 122 .

[0013] 3b is a diagram showing an example of the configuration of each pixel unit constituting the event-based sensor. The pixel unit 110 includes a photoelectric conversion unit 111, a pixel counter 112, a time counter 113, a first decision circuit 114, a memory 115, a comparator 116, a second decision circuit 117, a response circuit 118, and a selection circuit 119.

[0014] The photoelectric conversion unit 111 includes an avalanche photodiode (SPAD) that operates in Geiger mode, and is configured to count the number of photons that are incident on the photoelectric conversion unit 111 using a pixel counter 112. A time counter 113 counts the time that a photon is incident on the photoelectric conversion unit 111. By configuring an event-based sensor using a SPAD, it is possible to detect a luminance change at the level of one photon. By detecting a luminance change at the level of one photon, it is possible to acquire an address event signal even in a dark vision state such as at night.

[0015] When the number of photons counted by pixel counter 112 reaches a first threshold, first decision circuit 114 stops counting time by time counter 113. Past count values ​​of time counter 113 are stored in memory 115, and a comparator 116 is used to find the difference between the current count value of time counter 113 and the past count value of time counter 113.

[0016] If the difference count value is equal to or greater than the second threshold, the second determination circuit 117 sends a request signal to the vertical arbitration circuit 121 via the response circuit 118. The response circuit 118 receives a response from the vertical arbitration circuit 121 indicating whether output of the address event data is permitted or not. If the difference count value is less than the second threshold, the response circuit 118 does not send a request signal.

[0017] When the response circuit 118 receives a response indicating permission to output, the selection circuit 119 outputs the count value of the time counter circuit 113 to the horizontal output circuit 122. The horizontal output circuit 122 outputs the received count value as an output signal from the photoelectric conversion element 1011 to the detection unit 103.

[0018] Since the count value of the difference calculated by the comparator 116 corresponds to the inverse of the incidence frequency of photons, the photoelectric conversion element 1011 of the present invention has a function of measuring "changes in the incidence frequency of photons", that is, changes in luminance. In addition, the second judgment circuit 117 is used to output an address event only when the difference in the intervals at which the number of incident photons reaches the first threshold is equal to or greater than the second threshold. In other words, the photoelectric conversion element outputs the incidence frequency when the difference in incidence frequency is large, and does not output the incidence frequency when the difference is small. With the above configuration, an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time for each pixel address can be realized.

[0019] <Variations of photoelectric conversion elements> In the above, a case has been shown in which a photoelectric conversion element is used that uses a SPAD in the photoelectric conversion section and detects changes in the frequency of photon incidence by measuring the time at which a photon is incident. However, if an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time is used, the configuration does not have to be as shown in FIG. 2. For example, as described in Patent Document 1, a photoelectric conversion element that detects changes in luminance as voltage changes may be used.

[0020] <Image Generation Section> Based on the acquired address event signal, the image generating unit 202 generates a time-series image indicating the positions of at least one or more pixels where a change in luminance has occurred within a predetermined time range and the direction of the change in luminance. Specifically, the information on the luminance change detected by the event-based sensor is rearranged into a raster scan format that is easy to display on the display unit 104. That is, the image generating unit 202 generates an XYZ image by converting the address event signal into a coordinate space formed by a T axis (first axis) indicating the time axis and XY axes (second axes) related to space.

[0021] As described above, the signal output order from the photoelectric conversion element 1011 is controlled by the vertical arbitration circuit 121 in the photoelectric conversion element 1011, and the signals are output (in chronological order) in the order in which the luminance change of a threshold value or more occurs. Therefore, unlike synchronous photoelectric conversion elements such as general CMOS image sensors, the pixel output order is not in the frame format of raster scan. Therefore, in the image generation unit 202, the signals output from the photoelectric conversion element 1011 during a specific time range are temporarily stored in memory, and then the signals are rearranged for each pixel address to convert them into the frame format of raster scan. By repeating this conversion multiple times, an XYT image in a format having multiple frame images (XY images) in the time direction (T direction) can be generated.

[0022] If it is desired to display a phenomenon in real time, the specific time range may be set to the reciprocal of the refresh rate of the display unit 104. If it is desired to display a rapid change in a subject in slow motion, the specific time may be shortened according to the desired display speed. Alternatively, the user may input a designation via the user interface 105 and set the time range.

[0023] <Display section> The display unit 104 three-dimensionally displays the coordinates and time of pixels where a luminance change occurred. Fig. 4(a) is an example of an XYT image displayed by the display unit 104. White indicates that the luminance has changed in a positive direction, black indicates that the luminance has changed in a negative direction, and gray indicates that there has been no change in luminance (or that the luminance is less than the threshold value).

[0024] <Flowchart> The process executed by the information processing device 100 to display the address event signal described above will be described. FIG. 10 is a flowchart for describing the process executed by the information processing device 100. The process shown in the flowchart of FIG. 10 is executed by a computer, CPU 102, in accordance with a computer program stored in memory 103. In the following description, each process (step) is represented by adding an S to the beginning, and the representation of the process (step) will be omitted. However, the information processing device 100 does not necessarily have to perform all the steps described in this flowchart.

[0025] In S1001, the information processing device 100 initializes various settings. For example, a time range for measuring the number of occurrences of luminance changes is set. In addition, a threshold is set according to photon count noise. For example, the result of measuring the luminance change in a state where there is no moving object in the monitoring area is obtained, and a value at which an address event signal is observed is obtained. Since the address event signal observed at this time is likely to be noise, the number of occurrences of the noise is set as the threshold. The threshold may be set by other methods. An arbitrary value may be set by the user. Next, in S1002, the acquisition unit 201 acquires an address event signal output by the event-based sensor. This address event signal includes the address of the pixel where the luminance change occurred, the direction of the luminance change, and time information that can specify the time when the luminance change occurred. In S1003, the display control unit 203 determines whether the display method of the address event signal is to display in chronological order or in reverse chronological order. Here, the determination is made based on whether the display method input by the user to the operation unit 105 is chronological order. The display method may be determined according to a display method set in advance. For example, the time series may be switched between forward and reverse at predetermined intervals. When the display control unit 203 displays the images in chronological order, the process proceeds to S1004, and when the display control unit 203 displays the images in reverse chronological order, the process proceeds to S1005. In S1004, the image generating unit 202 generates a time series image indicating the positions of at least one or more pixels where a luminance change has occurred and the direction of the luminance change based on the address event signal. Here, in order to display an image by reversing the time series, an image is generated in which the direction of the luminance change indicated by the acquired address event signal is reversed. When a certain amount of address event signals are accumulated, the image generating unit 202 integrates the luminance change in a specific time range for each pixel. If there is a luminance change, the direction (positive or negative) and the amount of change are obtained. When the luminance change is to be displayed in reverse chronological order, the amount of change in luminance is left unchanged, and the direction of the luminance change is converted to the negative direction if it is in the positive direction, and converted to the positive direction if it is in the negative direction, to generate an image.In S1005, the image generating unit 202 generates a time-series image indicating the position of at least one or more pixels where a change in luminance has occurred and the direction of the change in luminance based on the address event signal. Here, in order to display in the forward direction of the time axis, the image is generated without changing the direction of the change in luminance. In S1006, the display control unit 203 causes the generated image to be displayed on the display unit in the set time-series. In S1007, the information processing device 100 determines whether or not to end the process. The end may be determined based on a user's end instruction, or may be determined based on whether or not all previously set processing targets have been processed. If the process does not end, the process returns to S1002.

[0026] <User interface> The user interface 105 is a part where the user controls the information processing device 100. Specifically, as shown in FIG. 4(b), the user interface 105 has a function of switching the bird's-eye view direction of the XYT image, enlarging or reducing the scale of each axis of the XYT image, and cutting out and displaying a part of the XYT image. In particular, in the present invention, the user interface 105 has a function of switching the playback direction of the XYT image displayed on the display unit 104 between a forward direction (the direction in which time advances) and a reverse direction (the direction in which time goes back). FIG. 11 illustrates an example of a user interface. G1100 is a graphic user interface (GUI). The user can check the display of the GUI screen and operate the display method of the address event signal using a touch panel or various input devices. In addition, the screen displays "playback" and "reverse playback" to clearly show the user the display method in which the image is displayed. G1101 is a screen that displays an image based on the address event signal according to this embodiment. Here, the address event signal is plotted on a three-dimensional coordinate space of the XYT axes, but each axis may be rotated and displayed, or an image generated at each time point may be displayed. G1102 is a button for changing the time axis to the opposite side or the front side of the viewpoint. G1103 is a button for rotating the spatial axis XY axis. G1104 is a button for adjusting the speed at which the generated image is played. G1105 is a button for initializing various settings. G1106 is a UI for specifying the time range of the displayed image. The user can specify the start point and the end point for generating the image by operating the arrow. G1107 is a reverse play button. It is selected when displaying the image in the reverse direction of the time axis. G1108 is a pause button. G1109 is a play button. It is selected when displaying the image in the forward direction of the time axis. The GUI is not limited to the form described here. The display unit 104 and the interface 105 may be provided in an external device connected to the information processing device 100 by a network cable or wireless transmission.

[0027] <Effects> In the information processing device according to the present embodiment, when the XYT image is played back in the reverse direction, the luminance change for each pixel is reversed and displayed. By using such a display, in a use case of detecting an abnormality in a subject that moves periodically at high speed, the possibility that the user can correctly recognize the occurring phenomenon can be improved. For example, FIG. 9(c) is an image in which each frame image of FIG. 9(b), which is a forward playback image, is rearranged in the reverse direction. Since the coordinate 1030 in FIG. 9(b) is displayed in black at time T0, it can be seen that the luminance has changed negatively at time T. In other words, if the actual luminance change is played back in the reverse direction, it should be displayed that the luminance has changed in the positive direction by more than the threshold value at time T. However, since the coordinate 1030 is displayed in black at time T0 in FIG. 9(c), a phenomenon in which the luminance has changed negatively by more than the threshold value is displayed. In other words, the luminance change expressed in FIG. 9(b) cannot be correctly played back in the reverse direction. Therefore, when images are played back in chronological order and when the images are played back in reverse, the direction of the brightness change that occurs in a certain pixel appears to be the same, so when played back in reverse, a phenomenon that actually occurred is displayed.

[0028] On the other hand, Fig. 4 shows a forward playback image and a reverse playback image according to this embodiment. Fig. 5(a) shows a cross section of the XYT image of Fig. 9(a) cut at Y=0, just like Fig. 9(b), and Fig. 5(b) shows the forward playback image shown in Fig. 5(a) converted into a reverse playback image using the method of this embodiment.

[0029] In Fig. 5(b), which is a reverse playback image according to this embodiment, the direction of the luminance change of the pixels in each frame of Fig. 5(a) is reversed and displayed. That is, the pixels expressed as white in Fig. 5(a) are converted to black in Fig. 5(b), and the pixels expressed as black in Fig. 5(a) are converted to white in Fig. 5(b).

[0030] Focus on coordinate 130 in FIG. 5(a), which corresponds to coordinate 1030 in FIG. 9(b). Since FIG. 5(a) and FIG. 9(b) are the same reproduced image, coordinate 130 in FIG. 5(a) is displayed in black at time T0, and it can be seen that the luminance has changed to a negative value at time T0. In other words, if the actual luminance change is reproduced in the reverse direction, it should be displayed as a luminance change in a positive direction at time T. On the other hand, coordinate 130 in FIG. 5(b) is displayed in white at time T0, and it is displayed as a luminance change in a positive direction at time T0. In other words, the reverse-reproduced image in FIG. 5(b) is a correct reverse reproduction of the forward-reproduced image in FIG. 5(a).

[0031] <Playback direction display> It is more preferable that whether forward playback or reverse playback is being performed is displayed on the display unit using text information or the like.

[0032] <Use Case> An information processing device using an event-based sensor may detect an abnormality occurring in a subject that moves periodically at high speed, such as an assembly robot in a factory. In such a use case, a user visually checks the XYT image to identify a time period in which an abnormality may occur, such as a part where the periodic movement of the subject is disturbed. Next, in order to analyze in detail the time period in which an abnormality may occur, the XYT image is repeatedly played forward (the direction in which time advances) and backward (the direction in which time goes back), and the presence or absence of an abnormality and its cause are identified. In this way, when the XYT image is played backward, the luminance change of each pixel is reversed and displayed. By using such a display, it is possible to improve the possibility that a user can correctly recognize the occurring phenomenon in a use case in which an abnormality occurring in a subject that moves periodically at high speed is detected.

[0033] <Embodiment 2> <Changing the display during reverse playback> In the information processing device 100 shown in the embodiment 2, the XYT image displayed during reverse playback is different. Specifically, since the display of an XYT image suitable for forward playback is different from the display of an XYT image suitable for reverse playback, an XYT image different from that displayed during forward playback is displayed during reverse playback.

[0034] <Change in perspective regarding the time axis> When displaying an image in which an address event signal is plotted in a coordinate space composed of a first axis indicating the time axis and a second axis related to space, the positions of the origin and the axes may be changed depending on the display method. FIG. 6 shows a display of an image in which the XYT image of FIG. 4(a) is played in the reverse direction. FIG. 6(a) is an XYT image seen from the same viewpoint as FIG. 4(a), and FIG. 6(b) is an XYT image in which the origin of the coordinates is changed so that the direction of the T axis (time axis) is reversed from that of FIG. 6(a). As can be seen from the figures, in FIG. 4(a) and FIG. 6(a), the past is displayed in the foreground and the future in the background, whereas in FIG. 6(b), the past is displayed in the background and the future in the foreground.

[0035] In the case of reverse playback, it is often used to estimate an abnormality of a phenomenon occurring in the future by going back to the past. Therefore, during reverse playback, it is preferable to display an XYT image in which the future is displayed in the foreground as shown in FIG. 6(b) rather than an XYT image in which the past is displayed in the foreground as shown in FIG. 6(a). Therefore, in the information processing device 100 shown in the second embodiment, during reverse playback, an XYT image with a changed viewpoint is displayed on the opposite side of the time axis from that during forward playback. In this way, by displaying an XYT image different from that during forward playback during reverse playback, visibility for the user is improved.

[0036] <Change of perspective on spatial axis> In addition to the time axis, with respect to the spatial axes (X-axis and Y-axis), even if the viewpoint is changed to the side opposite to the forward playback, during reverse playback, it is also possible to perform a display with the past on the back side and the future on the front side. However, it is better not to change the viewpoint with respect to the spatial axes and only change the viewpoint to the opposite side with respect to the time axis. This is because the correspondence between the XYT image during forward playback and the XYT image during reverse playback becomes easier, and when estimating the cause of an abnormality while repeating forward playback and reverse playback, a display without a sense of discomfort can be achieved.

[0037] <Cutout from XYT Image> Also, a use case where a part is cut out from the XYT image and displayed can be considered. For example, when it can be inferred that a phenomenon different from before and after occurs near a specific time in the XYT image, there may be a case where only the range near the specific time is extracted and displayed. As a display method in this case, two methods can be considered. One is the so-called normal moving image display that continuously updates the frame format (XY image) that displays the distribution information of the luminance change in the two-dimensional coordinate space in the spatial direction at a certain point in time over a specific period. The other is a display method in which a specific time is cut out from the XYT image and a plot representing the information of the luminance change is added in the image space where the XYT is displayed (hereinafter, plot-added XYT moving image).

[0038] In the information processing apparatus 100 shown in Embodiment 2, for each display method, a display suitable for each of forward playback and reverse playback is performed. The explanation will be given below. Hereinafter, the specific period to be cut out is assumed to be from time T1 (start point; first point) to time T2 (end point; second point).

[0039] <Frame Format Image> The image generating unit 202 generates a frame format image by plotting the change in luminance of each pixel, which is integrated in a specific time range at a certain time point, as distribution information in a two-dimensional coordinate space formed by an X axis and a Y axis. Furthermore, the display control unit 203 generates frame formats at multiple times. FIG. 7(a) is a frame format image generated from the forward playback image of FIG. 6(a), and FIG. 7(b) is a frame format image generated from the reverse playback image of FIG. 6(a). For the sake of explanation, FIG. 7 shows the direction in which the subject moves during forward and reverse playback with arrows. As can be seen from FIG. 7, the frame format image obtained by cutting out a part of the reverse playback image displays the luminance change for each pixel in reverse to that of the frame format image generated from the forward playback image.

[0040] That is, the frame format image generated from the reverse playback image is displayed with the luminance change of each pixel reversed, just like the reverse XYT playback image. This makes it easier for the user to recognize the phenomenon that is occurring, as the state of the luminance change of each pixel displayed in the XYT image, which provides an overview of the overall movement of the subject, and the frame format image, which displays detailed information of a portion of the XYT image, match.

[0041] <Future deletion display> A plot image is generated by plotting pixels in which a change in luminance occurs at a plurality of time points included in the specific period in a coordinate space formed by a first axis indicating a time axis and a second axis related to space. The generated plot images are displayed in a predetermined order by the display control unit 203. When displaying plot images in a reversed chronological order, all plot images generated in a specific period are displayed, and then the plot images generated at each time point are deleted in the order from the second time point to the first time point, and reverse playback is performed. When displaying plot images in chronological order, the plot images generated at each time point are superimposed and displayed in the order from the first time point to the second time point. FIG. 8(a) shows a plot-added XYT moving image generated from the forward playback image of FIG. 4(a), and FIG. 8(b) shows a plot-added XYT moving image generated from the reverse playback image of FIG. 6(a). Each of FIG. 8(a) and (b) shows a schematic representation in chronological order of how plots are added to represent a moving image.

[0042] 8(a) shows a moving image in which, starting from time T1, luminance change information is gradually added as a plot in the XYT space, and finally, at time T2, all luminance changes occurring within a specific time period from time T1 to time T2 are displayed on the XYT image. That is, this is a moving image in which luminance changes occurring in the future are gradually superimposed on luminance changes occurring in the past. By using such a plot-added XYT moving image, it is easy to grasp the causal relationship between a phenomenon occurring at a time of interest and a phenomenon occurring earlier than that, which is preferable.

[0043] On the other hand, Fig. 8(b) shows a moving image in which all luminance change information occurring from time T2 to time T1 is displayed on the XYT image, and when going back from the future to the past, plots of luminance change information occurring in the future are gradually deleted. That is, this is a moving image in which luminance changes occurring in the future than the current time are gradually deleted from an image in which all luminance changes occurring within a specific time are superimposed. By using such a plot-added XYT moving image, it is easier to grasp the causal relationship between a phenomenon occurring at a time of interest and a phenomenon occurring earlier during reverse playback, which is preferable.

[0044] That is, during forward playback, future luminance change information is gradually added to past luminance change information and displayed, and during reverse playback, future luminance change information is gradually deleted while past luminance change information is left. This configuration is preferable because it makes it easier to grasp the causal relationship between a phenomenon occurring at a time of interest and a phenomenon occurring earlier.

[0045] <Points to consider when displaying future deletion> In the case of a moving image in which luminance changes occurring in the future are gradually deleted as in Fig. 8(b), it is preferable to display the past in the background and the near side in the future. On the other hand, in the case of a moving image in which luminance changes occurring in the past are gradually added to luminance changes occurring in the future as in Fig. 8(c), it is preferable to display the past in the foreground and the future in the background. That is, during reverse playback, it is preferable to change the viewpoint on the time axis to the opposite side in the case where future luminance change information is gradually deleted while leaving past luminance change information intact, and in the case where luminance changes occurring in the past are gradually added to luminance changes occurring in the future.

[0046] <Display of past additions> FIG. 8(b) shows an example of displaying a moving image in which luminance changes occurring in the future are gradually deleted, so that it is easier to understand the causal relationship between a phenomenon occurring at a time of interest and a phenomenon occurring in the past. However, there may be a use case in which it is desired to look back in time to find the cause of a phenomenon occurring in the future. In this case, as shown in FIG. 8(c), it is preferable to display a moving image in which luminance change information is gradually added as a plot in the XYT space from time T2, and finally, at time T1, all luminance changes occurring within a specific time period from time T2 to time T1 are displayed on the XYT image. In other words, it is better to display a moving image in which luminance changes occurring in the past are gradually added to luminance changes occurring in the future.

[0047] <Other embodiments> The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or a CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium. [Explanation of symbols]

[0048] 100 Information processing device 101 Event-Based Sensors 201 Acquisition Department 202 Image Generation Unit 203 Display control unit 104 Display section 105 Operation section

Claims

1. an acquisition means for acquiring an address event signal indicating an address and a time of a pixel where a luminance change has occurred; generating means for generating a time-series image indicating the position of at least one pixel where a luminance change has occurred and the direction of the luminance change based on the address event signal; The information processing apparatus according to claim 1, wherein the generating means generates an image in which the direction of change in luminance is reversed when the image is displayed in a reversed time series.

2. 2 . The information processing apparatus according to claim 1 , wherein, when the image is displayed without changing the time series, the generating means generates the image without changing the direction of the change in luminance.

3. 3. The information processing apparatus according to claim 1, wherein the generating means generates the image by converting the address event signal into a coordinate space formed by a first axis indicating a time axis and a second axis relating to space.

4. 4. The information processing apparatus according to claim 3, wherein said generating means generates said image with the origin of said first axis on the opposite side when displaying said generated image in a reversed chronological order.

5. 5. The information processing apparatus according to claim 3, wherein the generating means generates the image with the origin of the second axis remaining unchanged when the generated image is displayed in a reversed chronological order.

6. 6. The information processing device according to claim 1, further comprising a control means for controlling a display method of the generated image based on the address event signal output during a specific period from a first time point, which is a start time point, to a second time point, which is an end time point.

7. 7. The information processing apparatus according to claim 6, wherein the control means displays different images when the generated images are displayed in reverse chronological order from when the generated images are displayed in chronological order.

8. the generating means generates a frame format image indicating a change in luminance of each pixel at each time point in a two-dimensional coordinate space formed by a second axis related to space, based on the address event signal; 8. The information processing apparatus according to claim 6, wherein the control means displays the frame format images at a plurality of points in time included in the specific period in a superimposed manner.

9. the generating means generates a plot image displaying pixels in which a luminance change occurs at a plurality of time points included in the specific period in a coordinate space formed by a first axis indicating a time axis and a second axis related to space; 8. The information processing apparatus according to claim 6, wherein the control means displays the generated plot images at a plurality of time points included in the specific period in a predetermined order.

10. The information processing device described in claim 9, characterized in that when the control means displays the plot images in reverse chronological order, it displays all of the plot images generated during the specific period and then deletes the plot images generated at each point in time in the order from the second point in time to the first point in time.

11. 11. The information processing apparatus according to claim 10, wherein the control means displays the first axis indicating the time axis with its origin on the opposite side.

12. The information processing device according to claim 9, characterized in that, when the control means displays the plot images in chronological order, the control means displays the plot images generated at each time point in superimposed order from the first time point to the second time point.

13. The information processing device according to claim 9, characterized in that, when the control means displays the plot images in reverse chronological order, the plot images generated at each time point are displayed superimposed in the order from the second time point to the first time point.

14. 13. The information processing apparatus according to claim 1, wherein the acquiring means acquires the address event signal based on an event-based sensor that outputs a signal in accordance with the number of incident photons.

15. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 14.

16. An acquisition step of acquiring an address event signal indicating an address and a time of a pixel where a luminance change has occurred; and generating a time-series image showing a luminance change and a direction of the luminance change in at least one pixel that occurred within a predetermined time range based on the address event signal, The information processing method according to the present invention, wherein, in the generating step, when the image is displayed in reversed time series, an image is generated in which the direction of the luminance change is reversed.

Images