Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program which facilitate abnormal detection of a subject in an event base sensor.SOLUTION: An imaging apparatus 100 comprises: an event base sensor which acquires an address event signal indicative of a pixel position and a time when luminance variation occurs; an evaluation unit which determines an evaluation value corresponding to frequency of the occurrence of the luminance variation in a predetermined direction for each pixel, on the basis of the address event signal; an image generation unit which generates an image indicative of the direction of the luminance variation, at the position of the pixel where the luminance variation occurs; and a display control unit which controls display of an image generated based on the evaluation value.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to the display of event-based sensors. [Background technology]

[0002] An event-based sensor is known that outputs a change in luminance of each pixel as an address event signal in real time (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-134271 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to facilitate the detection of an abnormality in a subject in an event-based sensor. [Means for solving the problem]

[0005] The information processing device of the present invention, which solves the above problem, is characterized by having an acquisition means for acquiring an address event signal indicating the position and time of a pixel where a luminance change has occurred, a determination means for determining an evaluation value corresponding to the frequency with which a luminance change has occurred in a predetermined direction for each pixel based on the address event signal, a generation means for generating an image indicating the direction of the luminance change at the position of the pixel where the luminance change has occurred, and a control means for controlling the display of the generated image based on the evaluation value. Effect of the Invention

[0006] This makes it easy to detect abnormalities in a subject using an event-based sensor. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 3] A diagram showing an example of the configuration of an event-based sensor [Figure 4] A diagram showing an example of a display [Diagram 5] A diagram showing an example of a display [Figure 6] A diagram showing an example of a display [Figure 7] A diagram showing an example of a display [Figure 8] A diagram showing an example of a user interface [Figure 9] A flowchart for explaining a process executed by an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] There is a method (hereinafter referred to as XY superimposed display) in which the event-based sensor displays a newly detected luminance change while retaining the display of past luminance changes. This XY superimposed display can particularly display the periodic movement of the subject in an easy-to-understand manner. This event-based sensor can be used to detect anomalies that occur infrequently in subjects that move periodically at high speed, such as machines used in the manufacturing industry. In such a use case, when a user visually checks a moving image captured by the event-based sensor and detects an abnormality, the user is likely to notice the abnormality because the display of past luminance changes including the abnormal operation remains. However, in practical terms, when the difference between normal operation and abnormal operation is small, it is difficult to distinguish the trajectory of normal operation from that of abnormal operation. Therefore, by using the information processing device according to this embodiment, it is possible to easily detect an abnormality in the subject in the measurement process using the event-based sensor. Hereinafter, the information processing device according to the embodiment of the present invention will be described with reference to the drawings. In this case, the same numbers are used for components having the same functions in all drawings, and repeated explanations are omitted.

[0009] <Embodiment 1> <Information processing device: Figure 1> FIG. 1 shows a block diagram illustrating an example of a hardware configuration of an information processing device. In FIG. 1, the information processing device 100 includes an event-based sensor 101 including an imaging optical system 1010 and a photoelectric conversion element 1011, a CPU 102, a memory 103, a display unit 104, and an operation unit 105. The event-based sensor 101 outputs an address event signal according to the received incident light. The imaging optical system 1010 is specifically a light receiving lens that receives incident light and forms an image on the photoelectric conversion element 1011. The photoelectric conversion element 1011 is specifically a SPAD sensor or a CMOS sensor that outputs an address event signal according to the received incident light. The CPU 102 reads out and executes an OS and other programs stored in the memory 103, controls each connected component, and performs calculations and logical judgments for various processes. The processes executed by the CPU 102 include the information processing according to this embodiment. Further, the CPU 102 controls the driving of the focus and aperture of the imaging optical system 1010, the driving of the photoelectric conversion element 1011, etc. The memory 103 is, for example, a hard disk drive or an external storage device, and stores programs and various data related to the information processing of the embodiment. The display unit 104 outputs the calculation results of the information processing device 100, etc. to a display device according to instructions from the CPU 102. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator. The operation unit 105 is, for example, a touch panel, a keyboard, a mouse, or a robot controller, and is a user interface that accepts input instructions from a user.

[0010] Next, an example of the functional configuration of the information processing device according to the present embodiment will be described with reference to FIG. 2. The information processing device 100 includes an event-based sensor 101, an acquisition unit 201, an image generation unit 202, a display control unit 203, an evaluation unit 204, a display unit 104, and an operation unit 105. Here, an overview of each function will be described. The event-based sensor 101 outputs an address event signal according to the received incident light. More specifically, the event-based sensor 101 outputs an address event signal indicating the address and time of a pixel where a change in luminance has occurred. The acquisition unit 201 acquires an address event signal indicating the position and time of a pixel where a change in luminance has occurred. The image generation unit 202 generates an image in which a predetermined pixel value according to the direction of the change in luminance is assigned to the position of the pixel where the change in luminance has occurred based on the address event signal. The display control unit 203 causes the display unit 104 to display the generated image. The evaluation unit 204 determines an evaluation value according to the frequency at which a change in luminance has occurred in a predetermined direction for each pixel. The display unit 104 displays the generated image. The operation unit 105 accepts an input by a user. The display unit 104 and the operation unit 105 may be realized by an external device of the information processing device. Each function will be described in detail below.

[0011] <Event-based sensor: Figure 3> An example of an event-based sensor according to the present embodiment will be described. The event-based sensor counts the number of incident photons and determines the timing when the number of counted photons exceeds a predetermined threshold. The event-based sensor also measures the time (clock count) required for the number of photons to reach or exceed a first threshold, and detects a change in luminance by comparing the required times. Specifically, when the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is equal to or greater than a second threshold, a change in luminance in the negative direction is detected. If the difference T0-T is equal to or greater than the second threshold, a change in luminance in the positive direction is detected. If the difference between T and T0 is less than the second threshold, no change in luminance is detected. The second threshold is a value equal to or greater than zero, and a value set in advance or according to other parameters is used.

[0012] 3a is a diagram showing an example of the configuration of a photoelectric conversion element 1011. The photoelectric conversion element 1011 is made up of a pixel section 110 and a peripheral circuit 120. The peripheral circuit 120 includes a vertical arbitration circuit 121 and a horizontal readout circuit 122.

[0013] FIG. 3b is a diagram showing an example of the configuration of each pixel unit constituting the event-based sensor. The pixel unit 110 includes a photoelectric conversion unit 111, a pixel counter 112, a time counter 113, a first judgment circuit 114, a memory 115, a comparator 116, a second judgment circuit 117, a response circuit 118, and a selection circuit 119. The photoelectric conversion unit 111 includes an avalanche photodiode (SPAD) that operates in Geiger mode, and is configured to count the number of photons incident on the photoelectric conversion unit 111 with the pixel counter 112. The time counter 113 counts the time during which the photon is incident on the photoelectric conversion unit 111. By configuring the event-based sensor using the SPAD, it is possible to detect a luminance change at the level of one photon. By detecting a luminance change at the level of one photon, it is possible to acquire an address event signal even in a dark vision state such as at night.

[0014] When the number of photons counted by pixel counter 112 reaches a first threshold, first decision circuit 114 stops counting time by time counter 113. Past count values ​​of time counter 113 are stored in memory 115, and a comparator 116 is used to find the difference between the current count value of time counter 113 and the past count value of time counter 113.

[0015] If the difference count value is equal to or greater than the second threshold, the second determination circuit 117 sends a request signal to the vertical arbitration circuit 121 via the response circuit 118. The response circuit 118 receives a response from the vertical arbitration circuit 121 indicating whether output of the address event data is permitted or not. If the difference count value is less than the second threshold, the response circuit 118 does not send a request signal.

[0016] When the response circuit 118 receives a response indicating permission to output, the selection circuit 119 outputs the count value of the time counter circuit 113 to the horizontal output circuit 122. The horizontal output circuit 122 outputs the received count value as an output signal from the photoelectric conversion element 1011 to the detection unit 103.

[0017] Since the count value of the difference calculated by the comparator 116 corresponds to the inverse of the incidence frequency of photons, the photoelectric conversion element 1011 according to this embodiment has a function of measuring "changes in the incidence frequency of photons", that is, changes in luminance. In addition, the second judgment circuit 117 is used to output an address event only when the difference in the interval of time at which the number of incident photons reaches the first threshold is equal to or greater than the second threshold. That is, the photoelectric conversion element outputs the incidence frequency when the difference in incidence frequency is equal to or greater than the second threshold, and does not output the incidence frequency when the difference is less than the threshold. With the above configuration, an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time for each pixel address can be realized.

[0018] <Variations of photoelectric conversion elements> In the above, a case has been shown in which a photoelectric conversion element is used that detects changes in the frequency of photon incidence by using a SPAD in the photoelectric conversion section and measuring the time at which a photon is incident. However, as long as the photoelectric conversion element detects changes in luminance as address events in real time, the configuration does not have to be as shown in FIG. 2. For example, as described in Patent Document 1, a photoelectric conversion element that detects changes in luminance as voltage changes may be used.

[0019] <Image Generation> The acquiring unit 201 acquires an address event signal indicating the position and time of a pixel where a change in luminance has occurred. Specifically, the position of a pixel where the luminance has changed in a positive or negative direction within a specific time range is acquired. The specific time range is a parameter that is set in advance, and may be a minimum time width determined by the time resolution of the vertical arbitration circuit 121 of the photoelectric conversion element 1011, or a longer time range may be specified by the user. A shorter time width allows for faster detection of an abnormal phenomenon, but a longer time width allows for reduced influence of errors due to random noise, etc., and improves the accuracy of distinguishing whether an abnormal phenomenon is occurring or not.

[0020] The image generating unit 202 generates an image that displays the direction of the luminance change in a predetermined direction from the address event signal. For example, a pixel value of 1 is assigned to pixels where a positive luminance change has occurred, and a pixel value of 0 is assigned to other pixels, thereby generating an image that represents only pixels where a positive trajectory change has occurred. Here, the pixel value of 1 may be any value other than 0. In addition, the color corresponding to pixel value 1 is described as "white" and the color corresponding to pixel value 0 is described as "gray", but other colors may be used. Here, the case where the luminance change is limited to positive is described, but the same effect can be obtained even when the luminance change is limited to negative. When the luminance change is limited to negative, in the following description, "positive luminance change" may be read as "negative luminance conversion" and "negative luminance change" as "positive luminance change". In addition, the display color "white" may be read as "black", and the display color "black" may be read as "white".

[0021] The evaluation unit 204 determines an evaluation value according to the frequency of occurrence of a luminance change in a predetermined direction for each pixel. The evaluation value is calculated from the frequency of positive luminance changes in the same pixel. Specifically, the difference ΔT in the timestamps between address events of positive luminance changes is taken, and the reciprocal of the timestamp difference, 1 / ΔT, is calculated. This is performed for each pixel in which a luminance change has occurred, and an evaluation value (frequency) for each pixel is calculated. It is desirable to suppress the influence of noise, etc. by taking the average value of the evaluation value with the previous value each time a positive luminance change occurs. Here, among the address events asynchronously sent from the photoelectric conversion element 1011, the positive luminance changes are used to calculate the evaluation value described later, and the negative luminance changes are discarded, but the evaluation value may be calculated according to the frequency of occurrence of the negative luminance changes.

[0022] The image generating unit 202 rearranges the information on the luminance change detected by the event-based sensor into a raster scan format that is easy for the display unit 104 to display. As described above, the signal output order from the photoelectric conversion element 1011 is controlled by the vertical arbitration circuit 121 in the photoelectric conversion element 1011, and the signals are output in the order in which the luminance change exceeding the threshold occurs. Therefore, unlike synchronous photoelectric conversion elements such as general CMOS image sensors, the pixel output order is not in the raster scan frame format. Therefore, the image generating unit 202 accumulates the signals output from the photoelectric conversion element 1011 during a specific time range in a memory, and then rearranges the signals for each pixel address to convert them into a raster scan frame format. If you want to display a phenomenon in real time, the specific time range can be the reciprocal of the refresh rate of the display unit 104. If you want to display a fast subject change in slow motion, you can shorten the specific time according to the speed you want to display.

[0023] The display control unit 203 controls the display of the image generated based on the evaluation value. That is, the display control unit 203 changes the way pixels are displayed depending on the evaluation value for each pixel calculated by the image generation unit 202. For pixels with evaluation values ​​smaller than a predetermined threshold (hereinafter referred to as the display threshold), a change in luminance in a predetermined direction is displayed in a superimposed manner. On the other hand, for pixels with evaluation values ​​equal to or greater than the display threshold, that is, pixels with a high frequency of positive luminance changes, the superimposed display is canceled and the display is returned to the default value of gray (the same display as pixels with no luminance change).

[0024] Here, for comparison, an example of XY superimposition display of a conventional method will be described. FIG. 4(a) shows the subject and the background. In the figure, 200 is the subject, and the white dotted line and white solid line are the trajectory of the subject 200 shown for explanation. In normal operation, the subject repeats circular motion along the white dotted line, and in abnormal operation that occurs infrequently, it passes through the path shown by the white solid line. In addition, the subject 200 has a high brightness compared to the background. FIG. 4(b) and (c) show examples of XY superimposition display of this subject photographed by an event-based sensor. FIG. 4(b) shows a display screen when no abnormal operation occurs and only normal operation is photographed. In this display screen, black indicates that a negative brightness change has been detected, white indicates that a positive brightness change has been detected, and gray indicates that there is no brightness change. Since the subject 200 is performing circular motion, a positive brightness change occurs in the area where the subject 200 has progressed in a specific time range (for example, the time corresponding to one frame), and the area becomes white. The area where the subject 200 has passed is displayed in black because a negative luminance change occurs, and since no new luminance change occurs until the subject 200 passes again, the display is not updated and remains as a black trajectory. FIG. 4(c) shows an example of an XY superimposed display when an abnormal operation occurs in addition to a normal operation, and the trajectory of the abnormal operation is added to FIG. 4(b). As can be seen by comparing FIG. 4(b) and (c), when the difference between the normal operation and the abnormal operation is small, the difference between the trajectories in the XY superimposed display image is also small, making it difficult to distinguish between the two. FIG. 5 shows an XY superimposed display image when the display method of this embodiment is used. The area surrounded by the dotted line (shown for explanation) in FIG. 5 is a trajectory part of a normal operation, so the evaluation value (frequency) is high, and it exceeds the display threshold, so the display is canceled. On the other hand, the white crescent-shaped area shown in the figure is a trajectory due to an abnormal operation. The evaluation value (occurrence frequency) of the abnormal operation part is low, so it is lower than the display threshold, and the trajectory remains displayed without disappearing. By displaying the screen as shown in Fig. 5, it is easy for the user to visually recognize that an abnormality has occurred. In other words, the trace of normal behavior, which occurs frequently, exceeds the threshold and is removed from the display, while the trace of abnormal behavior, which occurs infrequently, does not exceed the threshold and remains displayed, allowing the user to easily notice the occurrence of an abnormality.

[0025] <User interface> The user interface 105 is a part where the user controls the imaging device 100. Specifically, it has functions such as switching the direction (positive or negative) of the brightness change to be displayed, changing the value of the display threshold, changing the start time of the superimposed display, changing to another display method, and enlarging or reducing the scale of each axis of the XY image. In addition, the user may be allowed to select a part of the displayed image, and when selected, the display may be changed to an XY display (normal XY display, not a superimposed display) at the time when the trajectory of the selected area occurred. In this way, after the user finds an abnormality, the user can move on to analyzing the cause of the abnormality as soon as possible. The display unit 104 and the interface 105 described above may be provided in an external device connected to the imaging device 100 by a network cable or wireless transmission.

[0026] <Flowchart> The process executed by the information processing device 100 to display the address event signal described above will be described. FIG. 9 is a flowchart for describing the process executed by the information processing device 100. The process shown in the flowchart of FIG. 9 is executed by the CPU 102, which is a computer, in accordance with a computer program stored in the memory 103. In the following description, each process (step) is represented by adding an S to the beginning, and the representation of the process (step) will be omitted. However, the information processing device 100 does not necessarily have to perform all the steps described in this flowchart.

[0027] In S1001, the information processing device 100 initializes various settings. For example, a time range for measuring the number of occurrences of luminance changes is set. Also, a first threshold is set according to photon count noise. For example, the result of measuring the luminance change when there is no moving object in the monitoring area is obtained, and the value at which the address event signal is observed is obtained. Since the address event signal observed at this time is likely to be noise, the number of occurrences of the noise is set as the first threshold. The method of setting the first threshold may be other than the above. An arbitrary value may be set by the user. Next, in S1002, the acquisition unit 201 acquires an address event signal indicating the position and time of the pixel at which the luminance change occurred. This address event signal includes the address of the pixel at which the luminance change occurred, the direction of the luminance change, and time information that can specify the time at which the luminance change occurred. In S1003, the image generation unit 202 determines whether or not to display an image. If sufficient data for image generation has been collected (for example, address event signals have been acquired for a predetermined period or more) or if an output instruction has been issued by the user, the process proceeds to S1004 to generate an image. If the address event signals have not been collected sufficiently, the process returns to S1002. In S1004, the image generating unit 202 generates an image that displays the direction of the luminance change in a predetermined direction from the address event signal. In S1005, the evaluation unit 204 determines an evaluation value according to the frequency of occurrence of luminance changes in a predetermined direction for each pixel. In S1006, the display of the generated image is controlled based on the evaluation value. In S1007, the information processing device 100 determines whether or not to end the process. The determination may be based on a user's instruction to end the process, or may be based on whether or not all of the previously set processing targets have been processed. If the process does not end, the process returns to S1002.

[0028] <Evaluation value variations> The evaluation value mentioned above was "the frequency of luminance changes in the same direction," but as an extreme example, it may be "cancel the superimposed display if luminance changes in the same direction occur an even number of times, and turn on the superimposed display if they occur an odd number of times." This method requires less calculation and can be faster. With this method, the display of the normal operation trajectory is also turned on and off every time it occurs an even / odd number of times, so the display hunts (blinks). Since the frequency of occurrence of the abnormal operation trajectory is lower than the frequency of occurrence of the normal operation trajectory, the hunting interval of the abnormal operation trajectory is also longer than the hunting interval of the normal operation trajectory. For this reason, the user can visually distinguish abnormal operation from the difference in hunting intervals.

[0029] As another example of the evaluation value, the "number of times that luminance changes occurred in the same direction" may be used as the evaluation value. In a trajectory of normal operation, luminance changes in the same direction occur periodically, so the number of occurrences, i.e., the evaluation value, is high. On the other hand, in a trajectory of abnormal operation, luminance changes in the same direction occur infrequently because abnormal operation occurs infrequently, so the evaluation value is low. In this case, too, it is sufficient to cancel the superimposed display of pixels whose evaluation value exceeds the display threshold, just as in the case where frequency is used as the evaluation value. This method requires less calculations than using frequency as the evaluation value, and can be made faster.

[0030] <Display method variations> The display unit 104 may be configured to display parameters such as a display threshold value and a playback speed in text format to improve user convenience. Furthermore, the evaluation value of a pixel designated by the user via a user interface may be displayed in text format.

[0031] Alternatively, the overlay display of pixels with evaluation values ​​equal to or greater than the display threshold may be changed in color from pixels with evaluation values ​​less than the display threshold, so that the user can easily notice the abnormal behavior. Furthermore, evaluation values ​​equal to or less than the display threshold may be grouped into a predetermined range, and a different display color may be used for each group (e.g., divided into two groups). By displaying in this manner, when multiple types of abnormal behaviors with different occurrence frequencies are mixed, the possibility that the user will notice the presence of multiple abnormal behaviors can be improved.

[0032] <Variations in abnormality manifestations> Up to this point, the case where the frequency of occurrence of abnormal operations is lower than the frequency of occurrence of normal operations has been described. Here, two other patterns are assumed as abnormal occurrence patterns, and the effects of each will be described. The first is a case where the operation is normal at first, but becomes abnormal halfway through, and the abnormal operation is repeated thereafter. In this case, the frequency of occurrence of abnormal operations is high, so that the display of both the normal operation trajectory and the abnormal operation trajectory is eventually canceled. Since the abnormal operation trajectory does not remain even when the user looks at the displayed image, it is highly likely that the user will mistakenly think that everything is operating normally. In this case, it is preferable to display a warning such as "Possibility of abnormal operation occurring" on the display unit 104 to the user at the timing when the evaluation value for each pixel changes significantly (when the change in the evaluation value is greater than a predetermined ratio). This makes it possible to determine whether there is truly no change in luminance, even if nothing is displayed in the image, or whether it is not displayed because the frequency (evaluation value) of abnormal operations is high. Note that if the user gives an instruction via the user interface 105, the display may be changed to the XY display at the timing when the evaluation value for each pixel changes significantly. This enables users to quickly move on to analyzing the cause when an abnormality occurs.

[0033] The second case is where everything operates normally, but non-periodic operations are performed only at the initial startup stage and then enter periodic motion midway. In this case, the normal initial operation will be displayed in the same way as the abnormal operation trajectory. In this situation, the user can change the timing at which the user starts the XY overlay display via the user interface so that the display starts after the initial operation. Alternatively, when entering periodic motion, the user may reset the XY overlay display.

[0034] As described above, in the use case of detecting the occurrence of an abnormality in a subject that moves periodically at high speed, it becomes possible for the user to easily detect the occurrence of an abnormality.

[0035] <Embodiment 2> <XT display, YT display> In Embodiment 2, different from Embodiment 1, the overlay display (referred to as XT overlay display and YT overlay display) in the case of performing a two-dimensional space display using either one of the pixel address X and Y and the time axis T (XT display or YT display) will be described. Also, for the sake of explanation, the conventional display method without performing the overlay display is simply referred to as XT display and YT display. Hereinafter, the XT overlay display will be described as an example, but the same applies to the YT overlay display. In the case of the YT display, X and Y should be interchanged in the following description.

[0036] First, a conventional XT display (not a superimposed display) will be described. The XT display is known as a means for displaying changes in the time direction for the movement of a subject in an easy-to-understand manner. Consider the case where this display method is used to analyze a case where an object that periodically moves at a constant speed slows down due to an abnormality. FIG. 6 shows an example of a moving image in the conventional XT display. The subject is the same as that in FIG. 4(a). FIG. 6 shows changes in the moving image from time T1 (past) to T2 (future). The left side of the T axis represents the past, and newer events (X address values) are added to the right side of the T axis. In addition, the display range ΔT of the T axis is a fixed value, and after the time from the oldest data to the newest data exceeds ΔT, data up to ΔT before the newest data is displayed. In other words, past data that has passed a certain time ΔT or more from the newest data is no longer displayed. With such a display method, if there is an abnormality (speed change) in the periodic movement speed of the subject 200, there is a high possibility that the abnormal data will be overlooked by a person.

[0037] Next, the case of the XT superimposed display of this embodiment will be described. FIG. 7(a) shows a moving image image of the XT superimposed display. The photographed subject is the same as that of FIG. 4(a). In the XT superimposed display, as in the XY superimposed display of the first embodiment, only positive luminance changes are superimposed and negative luminance changes are not displayed. (On the other hand, the same effect can be obtained by displaying only negative luminance changes.) In addition, the evaluation unit 204 calculates an evaluation value (frequency) for each pixel, and the superimposed display of pixels whose evaluation value is equal to or greater than the display threshold value is canceled. That is, the image generating unit 202 generates an image showing the position of a pixel where a luminance change occurs at a predetermined time interval. Furthermore, the display control unit 203 superimposes and displays the images generated at a predetermined time interval. The evaluation unit 204 cancels the display of pixels (positions) where luminance changes occur in the same direction with a frequency above the predetermined display threshold value.

[0038] In Fig. 7(a), a specific time width ΔT is set as the display range. When the time exceeds ΔT from the start of shooting, redrawing is performed again from T = 0. Thereafter, the drawings from T = 0 are repeatedly and superimposed at intervals of ΔT. In this embodiment, it is assumed that the user adjusts the ΔT width to match one cycle of the subject 200 via the user interface 105. By setting ΔT in this way, the trajectories of each cycle overlap, so that the trajectory of normal operation becomes equal to or higher than the display threshold, and the superimposed display is cancelled. Fig. 7(b) shows a display image when an abnormal operation (cycle) occurs. The trajectory of normal operation exceeds the display threshold and the display is cancelled, and only the trajectory of abnormal operation is displayed. By looking at the image in Fig. 7(b), the user can easily notice the occurrence of abnormal operation.

[0039] Actually, even in a normal cycle, there is a certain degree of speed fluctuation, and there are cases where the trajectories do not completely overlap even when XT superimposed display is performed. In such a case, it can be solved by the user setting, via the user interface 105, a range in which the trajectories can be regarded as overlapping (referred to as the overlapping handling range). Fig. 7(c) shows the case where the overlapping handling range is set centered on the actual trajectory of normal operation. It is desirable to set the overlapping handling range to a value between the fluctuation width of normal operation and the deviation width of abnormal operation (the difference from the trajectory of normal operation), because only abnormal operation can be displayed while suppressing the influence of fluctuations.

[0040] <Embodiment 3> <XYT Display> In Embodiment 3, a method of three-dimensionally displaying the luminance change detected by an address event with the coordinates (XY) of the pixel where the event occurred and time (T) as axes (hereinafter referred to as XYT display) will be described. Since the imaging device in Embodiment 3 is the same as the imaging device 100 in Embodiment 1, descriptions of parts without differences will be omitted.

[0041] In this embodiment, the image generation unit 202 rearranges events that occurred during a specific time range by address and converts them into a raster scan frame format. By repeating this conversion multiple times, an XYT image can be generated that has multiple frame images (XY images) in the time direction (T direction). The display control unit 203 displays the generated XYT image three-dimensionally within the display area of ​​the two-dimensional screen. In other words, the display control unit 203 displays the generated image by changing the time axis and the space axis.

[0042] XYT display allows the recognition of the movement of a subject in three dimensions, making it suitable for analyzing cases where anomalies occur in both the spatial and temporal directions. It is also suitable as a display method for analyzing the cause after anomalies are discovered in the first and second embodiments.

[0043] Setting the viewpoint is important in XYT display; setting an inappropriate viewpoint will result in overlapping address events, i.e. an image with many blind spots. Figure 8(a) shows an example of an XYT display when a viewpoint with many blind spots is set. The subject is the same as in Figure 4(a). In the example shown in Figure 8(a), it is difficult to see the periodicity of the subject, especially in the time direction, which is undesirable as it makes it difficult for users to discover and analyze abnormal behavior.

[0044] In this embodiment, by setting a viewpoint that minimizes overlap between address events during XYT display, the user can easily find and analyze abnormal operations. In this embodiment, the image generating unit 202 calculates a viewpoint that minimizes overlap between address events. An example of viewpoint calculation will be described. The initial viewpoint is set to an angle on the T axis and viewing the XYT origin, and the position on the T axis is changed so that the entire address event fits within the display area of ​​the display system 104. It is desirable to determine the viewpoint position on the T axis so that the entire address event fits within the display area and data is placed near either end of the display area, since this reduces the count value. Next, the viewpoint is rotated around the X axis by a unit angle of 2π / n. This n (number of divisions) can be any value, but the smaller the value, the longer it takes to determine the viewpoint, and the larger the value, the lower the accuracy of the optimal viewpoint. After rotating the viewpoint, the number of address events that overlap on the display is counted and held. After this, the rotation of the viewpoint and the calculation of the count value are repeated until the total rotation angle becomes 2π, and finally the viewpoint angle with the smallest count value is set. Next, rotate the object around the Y axis by unit angle 2π / n, and calculate the count value for each angle in the same way. After repeating this process until the total rotation angle is 2π, the viewpoint with the smallest count value is determined to be the optimal viewpoint and adopted.

[0045] FIG. 8(b) shows an example of an XYT display displayed using the method of this embodiment. It can be seen that FIG. 8(b) has less overlap between address events compared to FIG. 8(a), making it easier to find abnormal behavior of the subject. As described above, the XYT display of this embodiment is suitable as a display for performing analysis after the user notices the occurrence of an abnormality in the first and second embodiments. In the first and second embodiments, when the user selects an abnormal trajectory via the user interface 105, the display may be changed to the XYT display at the timing when the corresponding trajectory occurred. This is desirable because it allows the user to move quickly to cause analysis after detecting an abnormal behavior.

[0046] <Other embodiments> The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or a CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium. [Explanation of symbols]

[0047] 100 Imaging device 101 Event-Based Sensors 104 Display section 105 Operation section 201 Acquisition Department 202 Image Generation Unit 203 Display control unit 204 Evaluation Department

Claims

1. acquiring means for acquiring an address event signal indicating a position and time of a pixel where a luminance change has occurred; a determining means for determining an evaluation value according to a frequency of occurrence of a luminance change in a predetermined direction for each pixel based on the address event signal; A generating means for generating an image showing a direction of the luminance change at a pixel position where the luminance change occurs; and a control means for controlling display of the generated image based on the evaluation value.

2. 2 . The information processing apparatus according to claim 1 , wherein the control means displays positions of pixels whose evaluation value is equal to or greater than a predetermined threshold value and positions of pixels whose evaluation value is less than the predetermined threshold value in a distinguished manner.

3. The information processing device according to claim 2, characterized in that the control means superimposes on the generated image an image in which a first pixel value indicating that no change in luminance has occurred in a predetermined direction is assigned to pixel positions where the evaluation value is equal to or greater than a predetermined threshold, and a second pixel value indicating that a change in luminance has occurred in a predetermined direction is assigned to pixel positions where the evaluation value is less than the predetermined threshold.

4. The information processing device according to claim 1 or 2, characterized in that the control means displays the generated image at pixel positions where the evaluation value is less than a predetermined threshold value, and does not display the generated image at pixel positions where the evaluation value is equal to or greater than a predetermined threshold value.

5. 5. The information processing apparatus according to claim 1, wherein the evaluation value is an integrated value of the number of occurrences of a change in luminance in a predetermined direction.

6. 6. The information processing apparatus according to claim 1, wherein the evaluation value indicates whether the number of occurrences of a luminance change in a predetermined direction is an even number or an odd number.

7. 6. The information processing apparatus according to claim 1, wherein the control means changes a display color of a pixel in accordance with a magnitude of the evaluation value.

8. the generating means generates an image indicating positions of pixels where a change in luminance occurs during a predetermined time interval; 8. The information processing device according to claim 1, wherein the control means displays the generated images so as to be superimposed at the predetermined time intervals in a two-dimensional space constituted by a spatial axis and a time axis.

9. 9. The information processing apparatus according to claim 1, wherein the control means outputs an abnormality notification when the evaluation value changes by a larger percentage than a predetermined percentage.

10. 10. The information processing apparatus according to claim 1, wherein the control means displays the image in a three-dimensional coordinate space defined by a T-axis indicating a time axis and X and Y-axis related to space.

11. 11. The information processing device according to claim 1, wherein the control means displays the image in a two-dimensional coordinate space defined by a T axis indicating a time axis and either an X or Y axis in space.

12. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 11.

13. acquiring an address event signal indicating the position and time of a pixel where a luminance change has occurred; determining an evaluation value according to a frequency of occurrence of a luminance change in a predetermined direction for each pixel based on the address event signal; A generating step of generating an image showing the direction of the luminance change at the position of the pixel where the luminance change occurs; and a control step of controlling display of the generated image based on the evaluation value.

Images