Information processing device, information processing method, and program

Abstract

The present disclosure pertains to an information processing device, an information processing method, and a program that make it possible to reduce the influence of system latency. On the basis of event data acquired by an EVS, a prediction model predicts the future states of the optical flow and the event data of a plurality of prediction widths corresponding to a plurality of latencies, and controls the operation on the basis of the prediction result. The present disclosure can be applied to a display device using the EVS.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable reduction in latency in operation control based on event data detected by an EVS (Event-based Vision Sensor).

[0002] In a display system in which imaging data is supplied from a terminal to a server, and a drawing is generated based on the imaging data at the server, and then the drawing is sent to the terminal for display, a technology has been proposed in which, in order to reduce communication latency, multiple drawings are generated from multiple position and orientation prediction results and selectively displayed (see Patent Document 1).

[0003] However, in the technology described in Patent Document 1, although latency related to communication is taken into consideration, so-called system latency related to image capture processing, calculation processing, and display processing is not taken into consideration.

[0004] Therefore, a technique has been proposed for reducing system latency by generating and displaying a predicted image taking system latency into consideration (see Patent Document 2).

[0005] JP 2021-114216 A JP 2020-048185 A

[0006] System latency changes dynamically depending on the ambient temperature and load conditions, but the technology described in Patent Document 2 is based on the premise that system latency is constant, and therefore may not be able to adequately reduce the impact of dynamically changing system latency.

[0007] The present disclosure has been made in consideration of such circumstances, and in particular, is intended to enable appropriate reduction of latency in operational control based on event data detected by an EVS (Event-based Vision Sensor) in consideration of dynamically changing system latency.

[0008] An information processing device and program according to one aspect of the present disclosure includes a first data acquisition unit that acquires first data, and a prediction unit that processes the first data and outputs multiple prediction results, wherein the prediction unit sets a prediction range that is a candidate value for system delay time and multiple times that include times different from the latest acquisition time of the first data as reference times, and predicts difference information from each of the reference times at times that are the prediction range ahead, starting from the latest acquisition time of the first data.

[0009] An information processing method according to one aspect of the present disclosure includes the steps of acquiring first data, processing the first data, and outputting a plurality of prediction results, in which a prediction range, which is a candidate value for system delay time, and a plurality of times including a time different from the latest acquisition time of the first data are set as reference times, and differential information from each of the reference times at a time that is the prediction range ahead, starting from the latest acquisition time of the first data, is predicted.

[0010] In one aspect of the present disclosure, first data is acquired, the first data is processed, and multiple prediction results are output; a prediction range, which is a candidate value for system delay time, and multiple times including times different from the latest acquisition time of the first data are set as reference times; and differential information from each of the reference times at times that are the prediction range ahead, starting from the latest acquisition time of the first data, is predicted.

[0011] 1. A diagram illustrating an example configuration of an EVS. 2. A diagram illustrating an example configuration of a pixel circuit of the EVS of FIG. 1. 3. A diagram illustrating the principle of latency generation and a method for reducing latency. 4. A diagram illustrating an example method for reducing latency. 5. A diagram illustrating dynamic changes in latency. 6. A diagram illustrating an overview of the principle of reducing dynamically changing latency. 7. A diagram illustrating an overview of the present disclosure for reducing dynamically changing latency. 8. A diagram illustrating an overview of the present disclosure for reducing dynamically changing latency. 9. A diagram illustrating an overview of the present disclosure for reducing dynamically changing latency. 10. A diagram illustrating an overview of the present disclosure for reducing dynamically changing latency. 11. A block diagram illustrating an example hardware configuration of an information processing device of the present disclosure. 12. A functional block diagram illustrating functions implemented by the information processing device of FIG. 10 in the first embodiment of the present disclosure. 13. A flowchart illustrating information processing by the information processing device of FIG. 14. 14. A diagram illustrating an overview of a second embodiment of the present disclosure. 15. A diagram illustrating an overview of a method for selecting event data in the second embodiment of the present disclosure. 18 is a functional block diagram illustrating functions realized by the information processing device of FIG. 10 in a second embodiment of the present disclosure. FIG. 19 is a flowchart illustrating information processing by the data processing block of the information processing device of FIG. 18. FIG. 20 is a flowchart illustrating information processing by the display processing block of the information processing device of FIG. 18. FIG. 21 is a diagram illustrating a specific method of selecting event data in the second embodiment. FIG. 22 is a diagram illustrating an overview of a third embodiment of the present disclosure. FIG. 23 is a diagram illustrating an overview of the third embodiment of the present disclosure. FIG. 24 is a diagram illustrating that in the third embodiment, an optical flow based on an RGB image is necessary. FIG. 25 is a diagram illustrating that in the third embodiment, an optical flow based on an RGB image is necessary. FIG. 26 is a diagram illustrating variations in optical flow. FIG. 27 is a diagram illustrating variations in optical flow. FIG. 28 is a diagram illustrating variations in the method of generating a predicted image. FIG. 29 is a diagram illustrating variations in the method of generating a predicted image.34 is a diagram illustrating a prediction result candidate group. FIG. 35 is a diagram illustrating a method for selecting one optical flow from a prediction result candidate group. FIG. 36 is a functional block diagram illustrating functions realized by the information processing device of FIG. 10 in a second embodiment of the present disclosure. FIG. 37 is a flowchart illustrating information processing by a data processing block of the information processing device of FIG. 32. FIG. 38 is a flowchart illustrating information processing by a display processing block of the information processing device of FIG. 32. FIG. 39 is a flowchart illustrating the prediction result selection process of FIG. 34. FIG. 39 is a diagram illustrating an example configuration of a general-purpose computer.

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] Hereinafter, embodiments of the present technology will be described in the following order.

[0014] 1. Overview of the present disclosure 2. First embodiment 3. Modification of the first embodiment 4. Second embodiment 5. Third embodiment 6. Example of execution by software

[0015] <<1. Overview of the Present Disclosure>> <Event Signal Detection Sensor> The present disclosure is particularly directed to making it possible to appropriately reduce latency in operation control based on event data detected by an Event-based Vision Sensor (EVS), taking into account dynamically changing system latency.

[0016] First, the EVS will be explained. Figure 1 shows an example of the configuration of the EVS.

[0017] 1 is configured as a pixel array in which a plurality of pixel circuits 11, each including a pixel 12 that performs photoelectric conversion of incident light to generate an electrical signal, are arranged in a lattice pattern on a two-dimensional plane. The EVS 1 performs photoelectric conversion of incident light in the pixel circuits 11 to generate an electrical signal, and generates and outputs event data that indicates the occurrence of an event, which is a change in the electrical signal.

[0018] Here, in the EVS 1, the portion where the plurality of pixel circuits 11 are arranged is a portion that receives incident light and performs photoelectric conversion, and is therefore also called a light receiving portion.

[0019] <Configuration Example of Pixel Circuit> FIG. 2 is a block diagram showing a configuration example of the pixel circuit 11 in FIG.

[0020] The pixel circuit 11 is composed of a pixel 12 and an event detection unit 13 .

[0021] The pixel 12 includes a photodiode (PD) 20 as a photoelectric conversion element. The pixel 12 receives light incident on the PD 20, performs photoelectric conversion, and generates and outputs a photocurrent (Iph) as an electric signal.

[0022] When a change exceeding a predetermined threshold (including a change equal to or greater than the threshold as necessary) occurs in the photocurrent generated by photoelectric conversion of the pixel 12, the event detection unit 13 detects the change in the photocurrent as an event. The event detection unit 13 outputs event data in response to (the detection of) an event.

[0023] Here, the change in the photocurrent generated in pixel 12 can also be considered as a change in the amount of light incident on pixel 12, so the event can also be said to be a change in the amount of light in pixel 12 (a change in the amount of light exceeding a threshold).

[0024] The event data can at least identify position information (coordinates, etc.) that indicates the position of the pixel 12 (pixel circuit 11) where a change in light amount as an event has occurred. In addition, the event data includes information that identifies a positive event or a negative event depending on the polarity (positive or negative) of the change in light amount.

[0025] For a series of event data output by the event detection unit 13 when an event is detected, time information indicating the (relative) time at which the event occurred can be identified as long as the interval between the event data is maintained as it was when the event occurred. However, if the interval between the event data is no longer maintained as it was when the event occurred, for example, due to the event data being stored in memory, the time information is lost. Therefore, time information indicating the (relative) time at which the event occurred, such as a timestamp, is added to the event data before the interval between the event data is no longer maintained as it was when the event occurred. The process of adding time information to the event data may be performed by the event detection unit 13 or external to the event detection unit 13, as long as it is performed before the interval between the event data is no longer maintained as it was when the event occurred.

[0026] The event detection unit 13 includes a current-voltage conversion unit 21 , a difference detection unit 22 , and a comparator 23 .

[0027] The current-voltage converter 21 converts the photocurrent from the pixel 12 into a voltage Vo (hereinafter also referred to as photovoltage) corresponding to the logarithm of the photocurrent, and outputs it to the difference detector 22 .

[0028] The difference detection unit 22 calculates the difference between the current optical voltage Vo from the current-voltage conversion unit 21 and the optical voltage at a timing that differs from the current by a small amount of time, and outputs the difference signal Vout to the comparator 23.

[0029] The comparator 23 compares the difference signal Vout output by the difference detection unit 22 with predetermined thresholds (voltages) +Vth and −Vth used for detecting an event. If the difference signal Vout is equal to or greater than the threshold +Vth or equal to or less than the threshold −Vth, the comparator 23 determines that a change in light amount as an event has been detected (occurred), and outputs event data.

[0030] For example, when the differential signal Vout is equal to or greater than the threshold +Vth, the comparator 23 determines that a positive event has been detected and outputs event data of a positive event. On the other hand, when the differential signal Vout is equal to or less than the threshold −Vth, the comparator 23 determines that a negative event has been detected and outputs event data of a negative event.

[0031] <Latency in a control device using an EVS> As described above, the EVS 1 detects a change in the amount of light received by a pixel 12 as an event, and outputs the coordinate position of the pixel at that time, time information (timestamp), and information indicating a positive event or a negative event as event data.

[0032] When the EVS 1 operates a device based on the event data output, if there is a latency with respect to the timing at which the event actually occurs, a delay occurs in the operation of the device with respect to the timing at which the event is detected.

[0033] As shown in the left part of FIG. 3, the latency mainly includes three types of latency: imaging latency, calculation latency, and control / output latency.

[0034] The imaging latency is the latency that occurs in the imaging system that outputs event data using the EVS 1 or the like.

[0035] The calculation latency is the latency that occurs in the calculation system that calculates various parameters required to control the operation based on the event data output from the EVS 1.

[0036] The control / output latency is the latency that occurs when an operation related to actual control or output is performed in a control system or output system (control system / output system) based on calculated parameters.

[0037] Hereinafter, the imaging latency, the calculation latency, and the control / output latency will also be collectively referred to as the system latency.

[0038] In FIG. 3 , the right direction is the time axis, and the top row shows the time series change in the vertical position of the object. The second row shows the timing at which event data is output from the imaging system. The third row shows the timing at which event data used in the calculation system is output. The fourth row shows the timing at which event data used in the control system / output system is output. The event data also indicates the position at which the event was detected in the entire pixel array when the event was detected.

[0039] That is, as shown in the left part of Figure 3, if no processing to reduce latency is performed, when an event is detected at event detection position coordinate P1 at time ts, event data V1 is output at time te due to the imaging latency in the imaging system.

[0040] The calculation system calculates parameters for controlling the operation of the control system / output system based on the event data V1 output from the imaging system at time te, resulting in calculation latency, and outputs parameters corresponding to the event data V1 at time tc.

[0041] The control system / output system will realize control and operation based on parameters corresponding to the event data V1 output at time tc in the calculation system, but at this time, due to control / output latency, the control and operation will be performed at time tm.

[0042] In other words, the action performed on the object at time tm is based on the event that occurred at time ts, and since the object is moving between time ts and time tm, for example, changing from position P1 to position P2, the action on the object is delayed due to the influence of the system latency LT (= tm - ts), and is not realized properly.

[0043] Therefore, as shown in the right part of Figure 3, at the time ts when an event is detected, based on the event data up to that point, including event data V1, a future object position P2 is predicted by a prediction model or the like for a time period corresponding to the system latency LT, and the event data V1 is corrected to event data V2 corresponding to the position P2, and then the processing of the calculation system and the operation of the control system / output system are realized, thereby reducing the influence of the system latency LT.

[0044] More specifically, when system latency is not taken into consideration, consider the case where event data Vi11 to Vi15 at event detection positions Pi11 to Pi15 in the real world are generated in chronological order at intervals equal to the system latency (e.g., 20 to 30 ms), as shown in the upper part of Figure 4.

[0045] At this time, the operation corresponding to the event detection position Pi11 of the event data Vi11 is performed by outputting the event data Vo21 of the event position Po21 corresponding to the event data Vi11 affected by the system latency at the same timing as the event data Vi12.

[0046] Similarly, the operations corresponding to the event detection positions Pi12 to Pi14 of the event data Vi12 to Vi14 are output as event data Vo22 to Vo24 of the event detection positions Po22 to Po24 affected by the system latency at the same timing as the event data Vi13 to Vi15, and are processed.

[0047] As a result, the operation based on the event data Vo21 to Vo24 is performed based on the event detection position of the previous event data delayed by the system latency, which causes a delay in the operation itself.

[0048] On the other hand, to reduce the effect of system latency, a prediction model is used to predict the event detection position Pr11 in the future by the amount of system latency based on the event detection position Pi11 (and the event data prior to that) in the event data Vi11, and event data Vr11 is generated, and operation is realized based on the event data Vo31 including the corresponding position Po31.

[0049] Similarly, based on the event detection positions Pi12 to Pi14 (and the event data prior to that) in the event data Vi12 to Vi14, future event detection positions Pr12 to Pr14 are predicted by the system latency using a prediction model, etc., to generate event data Vr12 to Vr14, and operation is realized based on the event data Vo32 to Vo34 including the corresponding positions Po32 to Vo34.

[0050] Furthermore, when predicting a future position by the system latency, a prediction delay occurs that is required for the prediction process, but since this is a time that is sufficiently short compared to the system latency, it is possible to sufficiently reduce the delay by generating event data that corresponds to the predicted position in the future by the system latency.

[0051] This process makes it possible to reduce the effect of system latency.

[0052] However, as shown in the left part of Figure 5, the system latency LT, which includes the image capture latency, calculation latency, and control / output latency from the time ts when an event is detected to the time tm when an operation is performed by the control system / output system, changes dynamically depending on the operating temperature and processing load, and is not a fixed time.

[0053] Depending on the operating temperature and processing load, for example, as shown in the upper right part of Figure 5, the system latency from the event detection time ts1 to the time tm1 may be LS (< TL), or as shown in the lower right part of Figure 5, the system latency may be LL (> TL) from the event detection time ts2 to the time tm2 when operation is performed.

[0054] That is, since the system latency is not a fixed time but changes dynamically, the effect of the latency cannot be reduced appropriately unless a correction is made according to the system latency at that time.

[0055] Therefore, it is conceivable to measure the system latency and then switch the prediction model depending on the measured system latency.

[0056] That is, as shown in the left part of Figure 6, for example, when the system latency LT is 20 ms and an event is detected at time ts11, a prediction model PM1 is prepared to predict the event detection position Po51 at time tm12, 20 ms in the future, based on input data Di consisting of event data up to that time, and event data Vo51 is generated.

[0057] Also, as shown in the right part of Figure 6, for example, when the system latency LT is 30 ms and an event is detected at time ts21, a prediction model PM2 is prepared to predict the position Po61 at time tm22, 30 ms in the future, based on input data Di consisting of event data up to that time, and event data Vo61 is generated.

[0058] However, preparing multiple prediction models according to system latency requires that the prediction model be trained for each system latency, which increases costs. In addition, since system latency can change dynamically even with the same hardware, switching prediction models each time is undesirable in terms of power consumption.

[0059] Therefore, in the present disclosure, as shown in Figure 7, a prediction model is prepared that predicts an event detection position corresponding to multiple system latencies based on multiple event data immediately before the timing at which the event is detected, and generates multiple event data that serve as prediction results, and selectively uses the prediction result corresponding to the system latency from among the multiple prediction results.

[0060] More specifically, the processing device illustrating the outline of the present disclosure in FIG. 7 includes a prediction model PM11, a latency setting unit LS, a selection unit SL, and a processing unit PS.

[0061] The prediction model PM11 assumes multiple times WP from the present to the future, assumed as system latency, for example, 10 (ms), 20 (ms), 30 (ms), and 40 (ms), from multiple event data linearly from the time the event is detected, and predicts each position Po81 to right Po84 (predicting difference information from each reference time at a time in the future by a prediction range based on the current time), and generates event data Vo81 to Vo84 according to the prediction results and outputs them to the selection unit SL.

[0062] In the following description, a plurality of times WP from the present time to the future that are assumed as system latency will also be referred to as a prediction width WP.

[0063] In addition, in this example, four types of prediction widths WP are set: 10 (ms), 20 (ms), 30 (ms), and 40 (ms), but other numbers of prediction widths may be set, and the time interval between multiple prediction widths may be other than 10 (ms).

[0064] The selection unit SL selects the event data VoSL of the corresponding predicted event detection position (predicted position) PoSL from the event data Vo81 to Vo84 according to the actual measurement value or the set system latency set by the latency setting unit LS, and outputs the selected event data to the processing unit PS.

[0065] In other words, the selection unit SL selects, from among the multiple event data Vo81 to Vo84, the event data of the event detection position predicted with a prediction width WP equivalent to the actual measurement value or the set system latency as the event data VoSL of the event detection position (predicted position) PoSL.

[0066] The processing unit PS executes control / output processing corresponding to the event data VoSL of the predicted position PoSL selected according to the system latency, and outputs the processing result VoP.

[0067] As a result, the prediction model PM11, which predicts the event detection position in the future based on the event data to be managed and generates event data based on the prediction results, is not configured for each prediction range but has a single configuration, thereby reducing management costs.

[0068] That is, for example, when the system latency is 20 ms, as shown in the left part of Figure 8, of the event data Vo81 to Vo84, each of which has a prediction width WP set to 10 (ms), 20 (ms), 30 (ms), or 40 (ms), the event data Vo82 at position Po82, where the prediction width WP = 20 corresponding to the system latency of 20 ms, is selected, and the corresponding processing is performed.

[0069] Furthermore, for example, if the system latency is 30 ms, as shown in the right part of Figure 8, of the event data Vo81 to Vo84, each of which has a prediction width WP set to 10 (ms), 20 (ms), 30 (ms), or 40 (ms), the event data Vo83 at position Po83, where the prediction width WP corresponds to 30 ms, is selected, and the corresponding processing is performed.

[0070] As a result, a prediction result according to the dynamically changing system latency is selected and processing is performed, so that the effect of latency can be reduced appropriately.

[0071] <<2. First embodiment>> <Outline of first embodiment> Next, an outline of a first embodiment of the present disclosure will be described with reference to Fig. 9. As shown in Fig. 9, the first embodiment of the present disclosure is an information processing device that executes a series of processes in which the trajectory of a ball B is detected as external world information by an EVS, and an arm robot RB swings a bat H to hit and return the ball B based on the detection result.

[0072] More specifically, the information processing device according to the first embodiment includes an EVS, a prediction model PM11, a latency setting unit LS, a selection unit SL, and a processing unit PS. The EVS, the prediction model PM11, the latency setting unit LS, the selection unit SL, and the processing unit PS basically correspond to the components in FIG. 7 .

[0073] The EVS is installed so that the trajectory of ball B is within its angle of view, detects the trajectory of ball B as an event, and outputs the event data to the prediction model PM11.

[0074] The prediction model PM11 is a prediction model that has been trained in advance by machine learning, such as a deep neural network (DNN), and predicts the event detection positions of the trajectory of the ball B for each of four prediction widths WP = 10, 20, 30, and 40 (ms), for example, based on a plurality of recent event data supplied from the EVS. The prediction model PM11 then generates event data Vo81 to Vo84 including the event detection positions that are the prediction results, and outputs them to the selection unit SL.

[0075] When the latency setting unit LS receives a system latency setting based on a user's operation input, the latency setting unit LS supplies information about the set system latency to the selection unit SL.

[0076] The selection unit SL selects a prediction result based on the prediction width corresponding to the set system latency from the event data Vo81 to Vo84 consisting of event detection positions with multiple prediction widths predicted by the prediction model PM11, and outputs it to the processing unit PS as event data VoP.

[0077] The processing unit PS recognizes the trajectory of the ball B taking into account the system latency based on the event data VoP in which the event detection position is predicted with a prediction width equivalent to the system latency, and operates the arm robot RB to hit the ball B with the bat H that it is holding.

[0078] The above series of processes realizes an operation that takes system latency into consideration, thereby making it possible to reduce the effects of latency.

[0079] <Detailed Configuration Example of Information Processing Device According to First Embodiment> Next, a configuration example of the information processing device according to the first embodiment of the present disclosure will be described with reference to the hardware block diagram of FIG.

[0080] The information processing device 31 in Figure 10 is composed of a control unit 41, an input unit 42, an output unit 43, a memory unit 44, a communication unit 45, a drive 46, a removable storage medium 47, an EVS 49, and an RGB camera 50, which are connected to each other via a bus 48 and can send and receive data and programs.

[0081] The control unit 41 is composed of a processor and a memory, and controls the overall operation of the information processing device 31. The control unit 41 also includes a system delay value setting unit 51, a data acquisition unit 52, a prediction unit 53, a prediction result selection unit 54, and a prediction result utilization unit 55.

[0082] In addition, the system delay value setting unit 51 to the prediction result utilization unit 55 are all programs pre-stored in a ROM (Read Only Memory) (not shown) or the storage unit 44, and are described as functioning when the control unit 41 reads them out, expands them into a RAM (Random Access Memory) (not shown), and executes them.

[0083] However, the system delay value setting unit 51 through the prediction result utilization unit 55 are not limited to those realized by software such as a program, and may have other configurations as long as they can realize the functions.

[0084] For example, each of the system delay value setting unit 51 to the prediction result utilization unit 55 may be configured and functioned by hardware such as a dedicated processor.

[0085] The functions realized by the system delay value setting unit 51 to the prediction result utilization unit 55 will be described in detail later with reference to FIG.

[0086] The input unit 42 is composed of input devices such as a keyboard, a mouse, and a touch panel through which the user inputs operation commands, and supplies various input signals to the control unit 41 .

[0087] The output unit 43 is controlled by the control unit 41 and includes a display unit and an audio output unit. The output unit 43 outputs and displays images of operation screens and processing results on the display unit, which is made up of a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The output unit 43 also controls the audio output unit, which is made up of an audio output device, to play various sounds, music, sound effects, and the like.

[0088] The storage unit 44 is composed of a hard disk drive (HDD), a solid state drive (SSD), or a semiconductor memory, and is controlled by the control unit 41 to write or read various data and programs including content data.

[0089] The communication unit 45 is controlled by the control unit 41, and realizes communication via wired or wireless means, such as LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs to and from various devices via a network (not shown) as necessary.

[0090] The drive 46 reads and writes data from and to a removable storage medium 47 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0091] The EVS 49 is the EVS 1 described with reference to Figures 1 and 2, and in this example, is installed in a direction that is within the angle of view when ball B arrives, and outputs the trajectory of ball B as event data.

[0092] The RGB camera 50 is configured with, for example, a CMOS image sensor, and captures and outputs an RGB image. As will be described in the following examples, the RGB camera 50 captures an image with a field of view that is substantially the same as the field of view of the EVS 49.

[0093] <Functions Realized by Information Processing Apparatus of First Embodiment> Next, functions realized by the information processing apparatus 31 of the first embodiment will be described with reference to the functional block diagram of FIG.

[0094] The functions realized by the information processing device 31 of the first embodiment are realized by the EVS 49, a system delay value setting unit 51, a data acquisition unit 52, a prediction unit 53, a prediction result selection unit 54, and a prediction result utilization unit 55.

[0095] The EVS 49 has a configuration corresponding to the imaging system described above, is set so that the trajectory of ball B is within the angle of view, detects the trajectory of the moving ball B as an event position where the brightness changes in the external world information, generates event data, and outputs it to the data acquisition unit 52.

[0096] The system delay value setting unit 51 has a configuration corresponding to the latency setting unit LS in Figure 9, and accepts the system delay value, i.e., the system latency setting, input by the user operating the input unit 42, and outputs it to the prediction result selection unit 54.

[0097] The data acquisition unit 52 acquires event data and RGB image data supplied from the EVS 49 and outputs them to the prediction unit 53 and the prediction result utilization unit 55. In addition, the data acquisition unit 52 acquires, as necessary, for example, RGB image data supplied from the RGB camera 50 and outputs it to the control unit 41 (the prediction result utilization unit 55 thereof).

[0098] The prediction unit 53 has a configuration corresponding to the prediction model PM11, and is composed of, for example, a DNN that has been trained in advance using machine learning, accumulates a plurality of most recent event data, predicts the trajectory of ball B, which is the event detection position for a plurality of prediction ranges, from the most recent plurality of event data, generates a plurality of event data including the event detection position that is the prediction result for each prediction range, and outputs the generated event data to the prediction result selection unit 54.

[0099] The prediction result selection unit 54 selects, from among the multiple event data, event data having a prediction range corresponding to the system latency based on the set system delay value, i.e., the system latency, supplied from the system delay value setting unit 51, and outputs the selected event data to the prediction result utilization unit 55.

[0100] The prediction result utilization unit 55 has a configuration equivalent to the control system / output system described above, and operates the arm robot 121 to swing the bat H and hit the ball B based on the event detection position in the future, i.e., the event data including the future trajectory, which is supplied by the prediction result selection unit 54 by the system latency.

[0101] Through the above series of processes, the event detection position is predicted for each of a plurality of prediction widths based on the event data generated by EVS 49, a plurality of corresponding event data are generated, and from among these, the event data corresponding to the system latency is selected, and the arm robot 121 is operated to hit the ball B with the bat H.

[0102] As a result, the operation of the arm robot 121 is controlled based on event data that includes information on the event detection position at a future time corresponding to the system latency, making it possible to reduce the effect of latency.

[0103] <Information Processing by Information Processing Device in FIG. 11> Next, information processing by the information processing device 31 in FIG. 11 will be described with reference to the flowchart in FIG.

[0104] In step S31, the system delay value setting unit 51 accepts a system delay value setting input by a user operating the input unit 42, and outputs the system delay value to the prediction result selection unit 54. The system delay value accepted here is a value that can be arbitrarily set by the user, but it is desirable that a value equivalent to a system latency measured in advance be input.

[0105] In step S32 , the data acquisition unit 52 acquires event data, which is supplied from the EVS 49 and in which the trajectory of the moving ball B is detected as an event position, for a predetermined period of time, and outputs the data to the prediction unit 53 .

[0106] In step S33, the prediction unit 53 performs preprocessing for the prediction process, such as by converting the event data for a predetermined time into frames of multiple channels for each prediction width, so that the data is in a processable format corresponding to the input layer of the prediction model consisting of a DNN.

[0107] In step S34, the prediction unit 53 predicts the trajectory of ball B, which is the event detection position of multiple prediction ranges, from the most recent multiple event data based on the preprocessed event data, generates multiple event data including the event detection position that is the prediction result for each prediction range, and outputs them to the prediction result selection unit 54.

[0108] In step S35, the prediction result selection unit 54 selects, from among the multiple event data, event data with a corresponding prediction range as a prediction result based on the set system delay value, i.e., system latency, supplied from the system delay value setting unit 51, and outputs the selected event data to the prediction result utilization unit 55.

[0109] In step S36, the prediction result utilization unit 55 operates the arm robot 121 to swing the bat H and hit the ball B based on the event data that takes into account the system latency supplied from the prediction result selection unit 54.

[0110] In step S37, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S32, and the subsequent processes are repeated.

[0111] Then, in step S37, if an instruction to end the process is given, the process ends.

[0112] Through the above processing, based on the event data generated by EVS 49, the event detection position is predicted for each of multiple prediction widths, multiple corresponding event data are generated, from which the event data corresponding to the system latency is selected, and based on the selected event data, the arm robot 121 operates to hit the ball B with the bat H.

[0113] As a result, the operation of the arm robot 121 is controlled based on event data that includes information on the event detection position at a future time corresponding to the system latency, making it possible to reduce the effect of latency.

[0114] <<3. Modification of First Embodiment>> <Outline of Modification of First Embodiment>> In the above, an example has been described in which a system delay value corresponding to a system latency is set by an input operation by the user.

[0115] However, the system latency may be measured in real time and the latency setting value corresponding to the system latency may be set based on the measurement result.

[0116] That is, as shown in Figure 13, the latency setting unit LS' measures the system latency from the imaging latency, calculation latency, and control / output latency in the actual operation of the imaging system, calculation system, and control / output system, and outputs the measured system latency to the selection unit SL.

[0117] More specifically, the imaging latency and the calculation latency change dynamically depending on the ambient temperature and the load state, but the control / output latency changes little depending on the ambient temperature and the load state, and is an almost fixed latency.

[0118] Therefore, in reality, the latency setting unit LS' measures the imaging latency and calculation latency of the actual imaging system and calculation system, respectively, and then estimates the system latency by adding a predetermined control / output latency, and outputs the estimated system latency (measurement result) to the selection unit SL.

[0119] The selection unit SL selects, based on the actually measured system latency supplied from the latency setting unit LS', event data including event position information of a plurality of predicted widths, event data including event position information of a predicted width corresponding to the system latency.

[0120] Note that Figure 13 shows a configuration in which only the components of the latency setting unit LS and the selection unit SL in Figure 7 are extracted, and further shows a configuration in which a latency setting unit LS' is provided instead of the latency setting unit LS.

[0121] The latency setting unit LS' has the same basic function as the latency setting unit LS. However, the latency setting unit LS differs in that, while the latency setting unit LS supplies the system latency input by the user to the selection unit SL, the latency setting unit LS' measures (estimates) the system latency from the imaging latency, calculation latency, and control / output latency in the actual operations of the imaging system, calculation system, and control / output system, respectively, and supplies the measured (estimated) system latency to the selection unit SL.

[0122] With this configuration, from among the event data containing event position information of a plurality of prediction widths, the event data corresponding to the dynamically changing system latency is selected, and processing is performed based on the selected event data.

[0123] As a result, it is possible to appropriately reduce the influence of dynamically changing system latency.

[0124] <Functions realized by the information processing device in the modified example of the first embodiment> Next, with reference to the functional block diagram of FIG. 14, functions realized by the information processing device 31 of FIG. 10 in the modified example of the first embodiment will be described.

[0125] In the information processing device 31 of FIG. 14, components having the same functions as those in the information processing device 31 of FIG. 11 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0126] The information processing device 31 in FIG. 14 differs from the information processing device 31 in FIG. 11 in that a system delay value setting section 51' is provided instead of the system delay value setting section 51.

[0127] The system delay value setting unit 51' has a configuration corresponding to the latency setting unit LS' in FIG. 13, and has the same basic function as the system delay value setting unit 51.

[0128] However, the system delay value setting unit 51' measures the imaging latency from when the event data output from the actual EVS 49 is generated until the event data is acquired by the data acquisition unit 52, and the calculation system latency from when the event data is acquired by the data acquisition unit 52 until the prediction unit 53 generates event data including event position information of multiple prediction widths.

[0129] Furthermore, the system delay value setting unit 51' measures (estimates) the dynamically changing system latency by adding predetermined control / output latency of the control system / output system, which changes little dynamically, to the measurement results of the imaging latency and the calculation system latency. As for the calculation system latency, since the calculation system latency cannot be measured in the subsequent prediction result utilization unit 55, it is assumed to be a predetermined latency and used for estimation.

[0130] Then, the system delay value setting unit 51 ′ sets a system delay value corresponding to the measured (estimated) system latency, and supplies the system delay value to the prediction result selection unit 54 .

[0131] With the above configuration, each time event data is generated in EVS 49, the system latency is measured (estimated), the event detection position is predicted for each of a plurality of prediction widths, a corresponding plurality of event data is generated, and from among these, the event data corresponding to the measured system latency is selected, and the arm robot 121 is operated to hit the ball B with the bat H.

[0132] As a result, the operation of the arm robot 121 is controlled based on event data containing information on the event detection position at a future time corresponding to the dynamically changing system latency, thereby making it possible to reduce the effects of latency.

[0133] <Information Processing by Information Processing Device in FIG. 14> Next, information processing by the information processing device 31 in FIG. 14 will be described with reference to the flowchart in FIG.

[0134] In step S51 , the data acquisition unit 52 acquires event data, which is supplied from the EVS 49 and in which the trajectory of the moving ball B is detected as an event position, for a predetermined period of time, and outputs the data to the prediction unit 53 .

[0135] In step S52, the prediction unit 53 performs preprocessing necessary for the prediction process by converting the event data for a predetermined time into frames of multiple channels for each prediction width, etc., so that the data can be processed by the input layer of the prediction model consisting of a DNN.

[0136] In step S53, the prediction unit 53 predicts the trajectory of ball B, which is the event detection position of multiple prediction ranges, from the most recent multiple event data based on the preprocessed event data, generates multiple event data including the event detection position that is the prediction result for each prediction range, and outputs them to the prediction result selection unit 54.

[0137] In step S54, the system delay value setting unit 51′ measures the imaging latency from when event data output from the actual EVS 49 is generated until the event data is acquired by the data acquiring unit 52, and the calculation system latency from when the event data is acquired by the data acquiring unit 52 until event data including event position information of a plurality of prediction widths is generated by the prediction unit 53. Then, the system delay value setting unit 51′ measures (estimates) the system latency by adding predetermined control / output latencies of the control system / output system to the measurement results of the imaging latency and the calculation system latency.

[0138] In step S55, the system delay value setting unit 51' sets the measured (estimated) system latency as a system delay value, and outputs it to the prediction result selection unit .

[0139] In step S56, the prediction result selection unit 54 selects event data of the corresponding prediction range from among the multiple event data based on the set system delay value, i.e., system latency, supplied from the system delay value setting unit 51′, and outputs the selected event data to the prediction result utilization unit 55.

[0140] In step S57, the prediction result utilization unit 55 operates the arm robot 121 to swing the bat H and hit the ball B based on the event data that takes into account the system latency supplied from the prediction result selection unit 54.

[0141] In step S58, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S51, and the subsequent processes are repeated.

[0142] Then, in step S58, if an instruction to end the process is given, the process ends.

[0143] Through the above processing, each time event data is generated by EVS 49, the system latency is measured (estimated), and the event detection position is predicted for each of multiple prediction widths, and multiple corresponding event data are generated. Of these, event data corresponding to the system latency is selected, and based on the selected event data, the arm robot 121 operates to hit the ball B with the bat H.

[0144] As a result, the operation of the arm robot 121 is controlled based on event data containing information on the event detection position at a future time corresponding to the dynamically changing system latency, thereby making it possible to reduce the impact of dynamically changing latency.

[0145] <<4. Second embodiment>> <Outline of second embodiment> In the above, an information processing device 31 has been described in which, from a group of event data consisting of predicted positions with multiple prediction ranges based on event data in which the trajectory of ball B is the event detection position, event data with a prediction range corresponding to a system delay value corresponding to system latency is selected, and ball B is hit with a bat H by an arm robot RB based on the selected event data.

[0146] However, the configuration corresponding to the control system / output system may be a configuration other than an arm robot, and for example, based on event data in which the trajectory of ball B is the event detection position, an effect may be superimposed on ball B in accordance with the trajectory of ball B using a display device such as a projector device.

[0147] 16 is a diagram illustrating an overview of an information processing device that detects the trajectory of ball B as external world information using an EVS and, based on the detection results, uses a display device PL consisting of a projector device to display an effect on ball B. In the information processing device of FIG. 16, components having the same functions as those of the information processing device in FIG. 9 are given the same reference numerals, and their description will be omitted where appropriate.

[0148] That is, the information processing device of Figure 16 differs from the information processing device of Figure 9 in that a selection unit SL' and a processing unit PS' are provided instead of the selection unit SL and the processing unit PS, and further a memory unit ME is newly provided.

[0149] The memory unit ME stores the event data Vo81 to Vo84, which include the event detection positions that are prediction results based on the most recent plurality of event data supplied to the EVS by the prediction model PM11, as event data groups G1 to Gn, respectively.

[0150] Incidentally, the event data groups G1 to Gn managed in the memory unit ME may be managed so that only the most recent n event data groups are stored, and when the number of event data groups reaches n or more, the oldest event data groups are sequentially deleted.

[0151] The selection unit SL' selects the most recent event data group G1 to Gn predicted by the prediction model PM11 and stored in the memory unit ME, and further selects a prediction result based on the prediction width corresponding to the set system latency from the event data Vo81 to Vo84 consisting of event detection positions with multiple prediction widths in the selected event data group, and outputs it to the processing unit PS' as event data VoP.

[0152] More specifically, consider the case where, for example, as shown in FIG. 17, event data is generated at 10 ms intervals, at times t0, t10, t20, t30, t40, t50, ..., and an event data group consisting of three event data sets with prediction widths of 10 ms, 20 ms, and 30 ms, respectively, is generated, and the system latency is set to 10 ms.

[0153] 17, event data Vo80-1 is generated at time t, and event data Vo81-1 with a prediction width of 10 ms, event data Vo82-1 with a prediction width of 20 ms, and event data Vo83-1 with a prediction width of 30 ms are predicted based on the event data Vo80-1. An event data group G1 consisting of the event data Vo80-1 to Vo83-1 is then stored in the memory unit ME.

[0154] Similarly, event data Vo80-2 is generated at time t+10, and event data Vo81-2 to Vo83-2 are predicted with prediction widths of 10 ms, 20 ms, and 30 ms, respectively, based on the event data Vo80-2. Then, an event data group G2 consisting of the event data Vo80-2 to Vo83-2 is stored in the memory unit ME.

[0155] Furthermore, event data Vo80-3 is generated at time t+20, and event data Vo81-3 to Vo83-3 are predicted with prediction widths of 10 ms, 20 ms, and 30 ms, respectively, based on the event data Vo80-3. Then, an event data group G3 consisting of the event data Vo80-3 to Vo83-3 is stored in the memory unit ME.

[0156] Furthermore, event data Vo80-4 is generated at time t+30, and event data Vo81-4 to Vo83-4 are predicted with prediction widths of 10 ms, 20 ms, and 30 ms, respectively, based on the event data Vo80-4. Then, an event data group G3 consisting of the event data Vo80-4 to Vo83-4 is stored in the memory unit ME.

[0157] Here, for example, at time t+30, the selection unit SL' reads the most recent event data group G4, and selects event data Vo81-4 from the event data Vo80-4 to Vo83-4 that make up the event data group G4, with a predicted width equivalent to the set system latency of 10 ms, and outputs it to the processing unit PS'.

[0158] Furthermore, for example, if a delay of more than the expected 10 ms occurs due to the influence of the ambient temperature or processing load of the prediction model PM11, and event data Vo80-4 is not generated at time t+30, and as a result, event data group G4 is not generated, the selection unit SL' reads out the immediately preceding event data group G3 as the most recent one.

[0159] In this case, however, it is necessary to take into consideration the system latency of 10 ms at time t+30.

[0160] Therefore, the selection unit SL' selects the event data Vo82-3, which has a predicted width equivalent to time t+40, from the event data Vo80-3 to Vo83-3 that make up the event data group G3, by adding 10 ms, which is the system latency set at the current time t+30, and outputs it to the processing unit PS'.

[0161] That is, in the second embodiment, compared to the process of controlling the movement of the arm robot RB so that the bat H hits the ball B at one point on the trajectory of the moving ball B, as in the first embodiment, it is necessary to continuously grasp the position of the moving ball B and continuously superimpose and display the effect image.

[0162] Therefore, if the dynamically changing system latency cannot be adequately addressed, there is a risk that the effect image will not be superimposed and displayed in the appropriate position.

[0163] Therefore, in the second embodiment of the present disclosure, event data is generated by predicting the event detection position with multiple prediction ranges from the event detection positions of continuously generated event data and multiple previous event data, and the event data is continuously and sequentially accumulated as an event data group consisting of event data with multiple prediction ranges.

[0164] This means that even if a delay longer than expected occurs in the generation of event data and the most recent group of event data cannot be obtained, the most recent group of event data accumulated up to that point can be used instead, thereby preventing errors related to the superimposed display of effect images due to delays in the generation of event data.

[0165] Furthermore, by selecting and substituting the most recent event data group from the accumulated event data group, there is no significant deviation in the predicted trajectory of ball B, which makes it possible to reduce fatal errors such as no effect image being superimposed on ball B at all.

[0166] <Functions Realized by Information Processing Device of Second Embodiment> Next, functions realized by the information processing device 31 of Fig. 10 will be described with reference to the functional block diagram of Fig. 18. Note that in the information processing device 31'' of Fig. 18, components having the same functions as those in the information processing device 31 of Fig. 11 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0167] The functions realized by the information processing device 31'' of the second embodiment include a data processing block 41a and a display processing block 41b realized by a control unit 41, and a prediction result memory unit 151 realized by a memory unit 44.

[0168] The data processing block 41a is composed of an EVS 49, a data acquisition unit 52, and a prediction unit 53, and acquires event data generated by the EVS 49. Based on the event data, event detection positions of multiple prediction widths are predicted, event data corresponding to each predicted position is generated, and the generated event data is stored in the prediction result memory unit 151 as an event data group.

[0169] The display processing block 41b accesses the prediction result memory unit 151, reads out the most recent group of event data, sets a system delay value based on the read out group of event data, selects event data based on the set system delay value, and controls the display device 161 based on the selected event data to realize an effect display on ball B.

[0170] More specifically, the display processing block 41 b is made up of a system delay value setting unit 51 ′, a prediction result selection unit 54 , and a prediction result utilization unit 55 .

[0171] The system delay value setting unit 51'' sets a system delay value in the same way as the system delay value setting unit 51, but it accesses the prediction result memory unit 151 at a predetermined refresh rate, reads out the most recent event data group, sets a system delay value based on the time information (timestamp) in the event data that make up the event data group, and outputs the system delay value to the prediction result selection unit 54.

[0172] Based on the system delay value supplied from the system delay value setting unit 51', the prediction result selection unit 54 selects event data that has been predicted with a prediction range equivalent to the system latency from the event data registered in the most recent event data group read from the prediction result memory unit 151, and outputs the selected event data to the prediction result utilization unit 55.

[0173] The prediction result utilization unit 55 identifies the trajectory of ball B based on the selected event data, and controls the display device 161 to superimpose an effect image on ball B on the identified trajectory.

[0174] With the above configuration, based on the event data generated by EVS 49, the event detection position is predicted for each of multiple prediction widths, multiple corresponding event data are generated, and the event data group is stored in the prediction result memory unit 151.

[0175] In addition, the most recent event data group stored in the prediction result memory unit 151 is read out at a predetermined refresh rate, and a system delay value is set based on the time information (timestamp) of the event data that constitutes the read event data group.

[0176] Furthermore, based on the set system delay value, event data predicted with a prediction width corresponding to the system latency is selected from the read event data group.

[0177] Then, the display on the display device 161 is controlled so that an effect image is superimposed on the ball B based on the selected event data.

[0178] As a result, the operation of the display device 161 is controlled based on event data containing information on the event detection position at a future time corresponding to the system latency, thereby making it possible to reduce the impact of latency on the effect display projected onto ball B.

[0179] Event data and event data that are prediction results for multiple prediction ranges based on the event data are successively accumulated in the memory unit 151 as event data groups, so even if the most recent event data group cannot be obtained for some reason, the most recent event data group can be used instead to continue processing, thereby preventing the occurrence of fatal errors such as losing track of ball B and making it possible to stably continue the superimposed display of effect images.

[0180] The processing performed by the data processing block 41 a and the processing performed by the display processing block 41 b are performed in parallel but are not necessarily synchronized. In particular, the processing performed by the display processing block 41 b is performed at a refresh rate of, for example, 10 ms intervals in the display device 161.

[0181] <Information Processing by Data Processing Block in Information Processing Device of FIG. 18> Next, information processing by the data processing block 41a of the information processing device 31 of FIG. 18 will be described with reference to the flowchart of FIG.

[0182] In step S71 , the data acquisition unit 52 acquires event data, which is supplied from the EVS 49 and in which the trajectory of the moving ball B is detected as an event position, for a predetermined period of time, and outputs the data to the prediction unit 53 .

[0183] In step S72, the prediction unit 53 performs preprocessing for the prediction process by converting the event data for a predetermined time into frames of multiple channels for each prediction width, and thereby making it into a format that can be processed as a prediction model consisting of a DNN.

[0184] In step S73, the prediction unit 53 predicts the trajectory of ball B, which is the event detection position for multiple prediction ranges, from the most recent multiple event data based on the preprocessed event data, and generates multiple event data including the event detection position that is the prediction result for each prediction range.

[0185] In step S74, the prediction unit 53 stores, in the prediction result storage unit 151 as an event data group, a plurality of event data sets including the event detection positions that are prediction results for each of a plurality of prediction widths.

[0186] In step S75, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S71, and the subsequent steps are repeated.

[0187] Then, in step S75, if an instruction to end the process is given, the process ends.

[0188] By the above processing, an event data group consisting of event data with a plurality of prediction widths is generated and sequentially stored repeatedly in the prediction result storage unit 151. Note that it is sufficient that the most recent predetermined number of event data groups are stored in the prediction result storage unit 151, and therefore, if the predetermined number is exceeded, the oldest event data groups may be gradually deleted and new event data groups may be stored.

[0189] <Information Processing by Display Processing Block in Information Processing Device of FIG. 18> Next, information processing by the data processing block 41a of the information processing device 31 of FIG. 18 will be described with reference to the flowchart of FIG.

[0190] In step S91, the prediction result selection unit 54 reads and acquires the most recent event data group from among the event data groups stored in the prediction result storage unit 151. At this time, the system delay value setting unit 51'' acquires the event data group read directly from the prediction result storage unit 151 or indirectly via the prediction result selection unit 54.

[0191] In step S92, the system delay value setting unit 51'' calculates (estimates) a system delay value based on the time information (timestamp) of the event data that is the starting point of the prediction that constitutes the event data group, for example, from the time lag with the current time, and outputs the system delay value to the prediction result selection unit 54.

[0192] In step S93, the prediction result selection unit 54 selects event data of the corresponding prediction range from the multiple event data that make up the event data group based on the set system delay value, i.e., system latency, supplied from the system delay value setting unit 51'', and outputs the selected event data to the prediction result utilization unit 55.

[0193] For example, consider the case where event data groups G1 to G4 shown in FIG. 21 exist, similar to the event data groups G1 to G4 in FIG. 17, and each event data is set with a prediction interval set at every 10 ms.

[0194] 21, event data Vo80-1 is generated at time t, and event data Vo81-1 with a prediction width of 10 ms, event data Vo82-1 with a prediction width of 20 ms, and event data Vo83-1 with a prediction width of 30 ms are predicted based on the event data Vo80-1. The event data Vo80-1 to Vo83-1 constitute an event data group G1.

[0195] Similarly, event data Vo80-2 is generated at time t+10, and event data Vo81-2 to Vo83-2 with prediction widths of 10 ms, 20 ms, and 30 ms are predicted based on the event data Vo80-2, and the event data Vo80-2 to Vo83-2 constitute event data group G2.

[0196] Furthermore, event data Vo80-3 is generated at time t+20, and event data Vo81-3 to Vo83-3 with prediction widths of 10 ms, 20 ms, and 30 ms are predicted based on the event data Vo80-3, and the event data Vo80-3 to Vo83-3 constitute event data group G3.

[0197] Furthermore, event data Vo80-4 is generated at time t+30, and event data Vo81-4 to Vo83-4 with prediction widths of 10 ms, 20 ms, and 30 ms are predicted based on the event data Vo80-4, and the event data Vo80-4 to Vo83-4 constitute event data group G4.

[0198] Here, for example, if event data group G4 exists as the most recent event data group, the prediction result selection unit 54 reads out the event data group G4 and calculates a system delay value LT4 estimated from the timestamp of event data Vo80-4 and the lag until the current time from the event data Vo80-4 to Vo83-4 that constitute the event data group G4.

[0199] At this time, when the system delay value LT4, which is the system latency, is 10 ms, the prediction result selection unit 54 selects the event data Vo81-4 with a prediction width WP=10 at time t+40, which is time t+30+LT4 (=t+30+10), and outputs it to the prediction result utilization unit 55.

[0200] Furthermore, for example, if a delay occurs due to the influence of the ambient temperature or processing load of the prediction unit 53, and event data Vo80-4 has not been generated and event data group G4 has not been acquired at time t+30, when event data group G3 exists, the prediction result selection unit 54 reads out the immediately preceding event data group G3 as the most recent event data.

[0201] In this case, the prediction result selection unit 54 calculates a system delay value LT3 estimated from the timestamp of the event data Vo80-3 among the event data Vo80-3 to Vo83-3 that make up the event data group G3 and the lag until the current time.

[0202] At this time, the timestamp of event data Vo80-3 in event data group G3 is time t+20. If we assume that the system delay value, which is the system latency, in event data group G4 is 10 ms, then under similar conditions, the system delay value LT3 will be 20 ms, which is 10 ms longer than the system latency LT4, which corresponds to the time between event data Vo80-3 and Vo80-4.

[0203] Therefore, in this case, the prediction result selection unit 54 selects the event data Vo82-3 with a prediction width = 20 at time t + 40 (= time t + 20 + 20), which is obtained by adding the system delay value LT3 = 20 ms to the timestamp of the event data Vo80-3, and outputs it to the prediction result utilization unit 55.

[0204] That is, as shown in Figure 21, the prediction result selection unit 54 sets a system delay value based on the timestamp of event data Vo80 from EVS49 stored at the beginning of the most recent event data group and the system latency estimated from the time lag to the current time, determines the event data to select from the event data group based on the system delay value, and outputs the determined event data to the prediction result utilization unit 55.

[0205] In step S94, the prediction result utilization unit 55 operates the display device 161 to superimpose an effect image on ball B based on the event data that takes into account the system latency supplied from the prediction result selection unit 54.

[0206] In step S95, it is determined whether or not an instruction to end the process has been given, and if an instruction to end the process has not been given, the process proceeds to step S96.

[0207] In step S96, it is determined whether a predetermined time has elapsed, and the same processing is repeated until the predetermined time has elapsed. Once the predetermined time has elapsed, the processing returns to step S91, and the subsequent processing is repeated.

[0208] Then, in step S95, if an instruction to end the process is given, the process ends.

[0209] With the above configuration, based on the event data generated by EVS 49, the event detection position is predicted for each of multiple prediction widths, and multiple corresponding event data are generated and stored in the prediction result memory unit 151 as an event data group.

[0210] In addition, the most recent event data group stored in the prediction result memory unit 151 is read out at a predetermined refresh rate, and a system delay value is set based on the time information (timestamp) of the event data that constitutes the read event data group.

[0211] Furthermore, based on the set system delay value, event data at an event detection position predicted with a prediction width equivalent to the system latency is selected from the read event data group.

[0212] Then, based on the selected event data, the display of the display device 161 is controlled so that the effect image is superimposed on the ball B, and the effect image is superimposed on the ball B.

[0213] As a result, the operation of the display device 161 is controlled based on event data containing information on the event detection position at a future time corresponding to the system latency, making it possible to reduce the impact of latency when superimposing an effect image on ball B.

[0214] In addition, at this time, multiple event data groups are accumulated in the prediction result memory unit 151, so even if a delay occurs in the generation of the event data group and the most recent event data group is not updated, it is possible to reduce the impact of the delay in the generation of the event data group by selecting the event data of the immediately previous event data group.

[0215] <<5. Third embodiment>> <Outline of third embodiment> In the above, an example has been described in which event detection positions with a plurality of prediction widths are predicted based on event data from EVS 49, and event data corresponding to the event detection positions that are the prediction results are generated to form an event data group, which is stored in the prediction result storage unit 151, and then the most recent event data group is read out at a predetermined refresh rate, a system delay value is determined from the read event data, event data with a prediction width that corresponds to the system latency is selected from the event data group based on the system delay value, and a predetermined effect image is superimposed on the ball based on the selected event data.

[0216] However, an RGB camera 50 may be installed at a position where the angle of view is similar to that of the EVS 49 that generates the event data, and the RGB image captured by the RGB camera 50 may be warped based on the event data to generate and display a future RGB image corresponding to the system latency as a predicted image.

[0217] In this case, as shown in the top row of Figure 22, the RGB images captured by the RGB camera 50 have a low frame rate compared to the frame rate of the event data generated by the EVS 49, so the information is sparse compared to the density in the time direction at which the event data is generated.

[0218] That is, in the top row of Figure 22, the event data generated by EVS 49 is generated at a frequency shown by the dotted line, whereas the RGB images captured by RGB camera 50 are sparser and less frequent than the event data, as shown, for example, in frames F1 to F5.

[0219] The RGB images captured by the RGB camera 50 are generally captured at 30 fps, whereas the event data can be generated at a frame rate several to several hundred times higher than the RGB image frame rate.

[0220] Therefore, in this example as well, when an event is detected at event detection position P101 at time t and event data V101 is generated, it is necessary to predict the future event detection position P102 by the system latency, which is made up of imaging latency, calculation latency, and control / output latency, generate event data V102 corresponding to the predicted position, and then warp the pixels of frame F3 of the RGB image to generate a predicted image and display it in a superimposed manner.

[0221] However, in the third embodiment, unlike the previous embodiments, it is necessary to warp all pixels in the reference RGB image, so a predicted image cannot be generated by simply generating event data with a predetermined prediction width.

[0222] Therefore, in the present disclosure, the optical flow of all pixels is predicted with multiple prediction widths based on the most recent multiple event data corresponding to the angle of view of the RGB image, and the optical flow with the corresponding prediction width is selected based on the system delay value set corresponding to the system latency.

[0223] Then, by using the selected optical flow to warp each pixel of the reference RGB image, a future RGB image corresponding to the system latency is generated as a predicted image and displayed.

[0224] Here, optical flow is information made up of vectors indicating the direction and amount of movement of each pixel between two frames of images arranged in time series, or the motion vector of a subject object.

[0225] Therefore, once an optical flow of a predetermined prediction width is obtained, it is possible to generate a predicted image of the predetermined prediction width by moving (warping) the pixel values ​​of each pixel of the reference RGB image at the start of prediction according to a vector that uses each pixel as a unit set in the optical flow of the predetermined prediction width.

[0226] 23 illustrates an overview of an information processing device according to a third embodiment. In the third embodiment of the present disclosure, as illustrated in FIG. 23 , external world information is detected by an EVS, and an RGB camera 50 captures an RGB image with approximately the same angle of view as the EVS. Based on the detection results of the EVS, a group of optical flows consisting of optical flows with multiple prediction widths is predicted using a prediction model. Then, an optical flow with a prediction width corresponding to the system latency is selected from the group of optical flows. Based on the selected optical flow, each pixel of the RGB image FA captured by the RGB camera 50 is warped, and an RGB image FB that is future by the system latency is generated and displayed as a predicted image.

[0227] More specifically, the information processing device according to the third embodiment includes an EVS, an RGB camera, a prediction model PM31, a latency setting unit LS', a storage unit ME', a selection unit SL'', and a processing unit PS'.

[0228] The EVS is installed so that it has almost the same angle of view as the RGB camera. For example, as shown in Figure 23, it captures an image of a dancing person, detects changes in the amount of light received in the image as events, and outputs them to the prediction model PM31 as event data.

[0229] The prediction model PM31 is a prediction model that has been trained in advance by machine learning, such as a deep neural network (DNN), and predicts optical flows OF1 to OF4 for each of four prediction widths WP = 10, 20, 30, and 40 (ms) based on the most recent event data supplied from the EVS. The prediction model PM31 then assigns a timestamp to each of the optical flows OF1 to OF4 that are the prediction results based on the event data, and stores them in the memory unit ME' as an optical flow group (OF group).

[0230] In FIG. 23, the storage unit ME' stores OFG1 to OFGn.

[0231] The RGB camera is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures an image with approximately the same angle of view as the EVS as an RGB image at a frame rate of about 30 fps, and outputs it to the processing unit PS'.

[0232] The latency setting unit LS' estimates the system latency based on the timestamp of the latest optical flow stored in the memory unit ME', and sets a system delay value based on the estimation result, and supplies it to the selection unit SL''.

[0233] The selection unit SL'' reads out the latest OF group from the OF group predicted by the prediction model PM31, selects a corresponding optical flow from the optical flows of multiple prediction widths included in the read OF group based on the set system latency and the acquisition timing of the RGB image FA from the RGB camera, and outputs it to the processing unit PS' as an optical flow OFSL.

[0234] The processing unit PS' generates and displays a future RGB image FB corresponding to the system latency as a predicted image by warping each pixel of the RGB image FA based on the optical flow OFSL selected based on the timing at which the RGB image was acquired and the system latency.

[0235] Through the above series of processes, an optical flow is selected based on the system latency and the RGB image that serves as the basis for warping, the RGB image is warped using the selected optical flow, and an RGB image from the future by the system latency is generated and displayed as a predicted image, making it possible to display an RGB image while reducing the effects of latency that occur when capturing and displaying images with an RGB camera.

[0236] <Estimation of optical flow taking into account system latency and timing of acquisition of RGB image> In the third embodiment, it is necessary to be able to select an optical flow taking into account the system latency in terms of the timing of predicting the optical flow based on event data and the timing of acquisition of the RGB image that serves as the basis for warping.

[0237] In the first and second embodiments, event detection positions are predicted for multiple prediction widths based on event data, and event data corresponding to each prediction result is generated to form an event data group, and event data from the future by the system latency is selected.

[0238] However, as described above, the frame rate of the RGB image is sufficiently small compared to the frame rate at which event data is detected, so the RGB image is sparse with respect to the optical flow.

[0239] Therefore, in the third embodiment, if optical flows with multiple prediction widths are predicted based on event data to form a group of optical flows, and an optical flow is simply selected that is in the future by the system latency, and then an attempt is made to warp an RGB image, the RGB image that serves as the basis for warping will not necessarily exist at the prediction start time of the selected optical flow, and therefore warping may not be possible, and an RGB image as a predicted image may not be generated.

[0240] For example, as shown in the top row of Figure 24, consider the case where an RGB image of frame F3 is acquired at time t, and at time t + 10 (ms), optical flows with multiple prediction widths are predicted and generated, and an optical flow with a prediction width according to the system latency is selected.

[0241] In this case, in the first and second embodiments, for example, if the prediction width WP is 10, 20, 30, or 40 (ms), optical flow OF1 from time t+10 to time t+20, optical flow OF2 from time t+10 to time t+30, optical flow OF3 from time t+10 to time t+40, and optical flow OF4 from time t+10 to time t+50 are generated as prediction results, and optical flow groups are formed and stored.

[0242] When the system latency is 10 ms, the prediction width WP=10, and optical flow OF1 is selected to generate a predicted image at time t+20, which is the system latency from time t+10.

[0243] However, in the case of Figure 24, the RGB image of frame F3, which serves as the basis for warping, is acquired at time t. Therefore, if we try to warp the image using optical flow OF1 to generate an RGB image at time t+20, the interval between the two is 20 ms, so optical flow OF1 with a prediction width of WP=10 will not be able to perform appropriate warping, and an appropriate predicted image will not be able to be generated.

[0244] In the third embodiment, it is necessary to generate a predicted image at time t+10 using the RGB image acquired at time t, which is the most recent, as a reference.

[0245] Therefore, when warping an RGB image to generate a predicted image at time t+10, it is necessary to select an optical flow based on time t, which is the acquisition timing of the RGB image of frame F3 acquired at the most recent time t, as shown in Figure 25.

[0246] That is, at time t+10, for example, if the prediction width WP is 10, 20, 30, or 40 (ms), as shown in Figure 25, optical flow OF1' from time t to time t+20, optical flow OF2' from time t to time t+30, optical flow OF3' from time t+10 to time t+40, and optical flow OF4' from time t+10 to time t+50 must be generated as prediction results, and a group of optical flows must be formed.

[0247] Then, when the system latency is 10 ms, the optical flow OF1' from time t to time t+20 is selected, and each pixel of the RGB image of frame F3 at time t is warped using the selected optical flow OF1', thereby generating an RGB image at time t+10 that is a predicted image 10 ms into the future, which corresponds to the system latency.

[0248] Similarly, when an optical flow with a prediction width WP of 20 ms is assumed at time t+10, an optical flow OF11 from time t to time t+30 is required, as shown in the upper left of FIG.

[0249] Similarly, when an optical flow with a prediction width WP of 30 ms is assumed at time t+10, an optical flow OF12 from time t to time t+40 is required, as shown in the upper right part of FIG.

[0250] Furthermore, at time t+20, an optical flow based on the acquisition timing of the most recently acquired RGB image is also required.

[0251] Furthermore, since times t+10 and t+20 were not the timings for acquiring RGB images, an optical flow of a prediction width obtained by adding the time until the most recent RGB image was acquired to the prediction width WP at each timing would be required. However, times t and t+30, for example, coincide with the timings at which RGB images are acquired, so in this case, it is sufficient to use the optical flow of the required prediction width itself.

[0252] That is, when an optical flow with a prediction width WP of 20 ms is assumed at time t, an optical flow OF21 from time t to time t+20 is required, as shown in the lower left part of FIG.

[0253] Similarly, when an optical flow with a prediction width WP of 30 ms is assumed at time t, an optical flow OF22 from time t to time t+30 is required, as shown in the lower right part of FIG.

[0254] To summarize, as shown in Figure 27, the memory unit ME' needs to contain a mixture of optical flows required at times t+10, t+20, etc., which are different from the timing at which the RGB image is acquired, as described with reference to Figure 26, and optical flows required at times t, t+30, etc., which are the same as the timing at which the RGB image is acquired.

[0255] In addition, in Figure 27, the memory unit ME' represents a state in which optical flows OF11 and OF12 required at time t+10 and optical flows OF21 and OF22 required at time t are mixed together.

[0256] Optical flows OF21 and OF22 are assumed to correspond to the left and right optical flows in the lower part of FIG. 26, respectively, and optical flows OF11 and OF12 are assumed to correspond to the left and right optical flows in the upper part of FIG.

[0257] In addition, in the case of optical flow OF12 in Figure 27, which is shown in the upper right part of Figure 26, as shown in the left part of Figure 28, the RGB image that will be the future predicted image at time t+40, which is 30 ms after time t+10, is generated by warping the RGB image of frame F3, which is acquired at time t, 40 ms before time t+40, with optical flow OF12.

[0258] Therefore, in the case where an RGB image of a further updated frame F4 has already been acquired at time t+30 before the predicted image is displayed, the RGB image of frame F4 may be warped using optical flow OF12' from time t+30 to time t+40, as shown in the right part of Figure 28, so that a future RGB image at time t+40, 30 ms after time t10, is generated as the predicted image.

[0259] More specifically, for example, from the timing of time t+10, processing is performed according to the time schedule shown in FIG.

[0260] Between times t+10 and t+11, event data is generated by the EVS.

[0261] From time t+11 to t+21, optical flow (OF) is predicted for multiple prediction ranges based on the event data.

[0262] Between time t+21 and t+22, an optical flow is selected from the optical flows (OF) with a plurality of prediction widths based on a system delay value corresponding to the system latency and the timing of acquisition of the reference RGB image.

[0263] At times t+22 to t+25, each pixel of the reference RGB image is warped using the selected optical flow to generate an RGB image that is a future predicted image at time t+40.

[0264] Then, from time t+25 to time t+35, the generated RGB image, which is a predicted image for the future, is displayed.

[0265] That is, one display process is performed approximately 25 ms after the EVS starts acquiring event data.

[0266] In the case of the left part of FIG. 28, the RGB image of frame F3 is warped by the optical flow OF12 to generate an RGB image that is a future predicted image at time t+40.

[0267] Furthermore, the RGB image that will become frame F4 will be acquired at time t+30, but if, as shown in the lower left of Figure 29, the RGB image that will become frame F4 is captured while optical flows of multiple prediction widths are predicted and acquired by time t+21, then, as shown in the right part of Figure 28, the RGB image of frame F4 may be warped by optical flow OF12' to generate an RGB image that is a future predicted image at time t+40.

[0268] In this way, by generating a predicted image using frame F4, it is possible to perform warping using a more recent image than when a predicted image is generated using frame F3, making it possible to generate a predicted image with high prediction accuracy.

[0269] When generating a predicted image in this way, there are multiple candidates for the optical flow used to warp the reference RGB image, based on the relationship between the reference time, which is the time when the RGB image used as the reference for warping is acquired, and the system latency.

[0270] <Variations of Optical Flow> For example, if RGB images captured by an RGB camera are acquired at intervals of 30 ms, event data is acquired at intervals of 10 ms, and optical flows with prediction widths WP of 10 ms, 20 ms, 30 ms, and 40 ms are generated, the optical flows that generate predicted images will have multiple variations depending on the timing at which the reference RGB images used for warping are acquired, even for the same system latency.

[0271] In other words, when the system latency is 10 ms, if the reference time, which is the time when the reference RGB image was acquired, is the same as the predicted start time, an optical flow is used that predicts an image 10 ms in the future relative to the predicted start time, which is the same as the reference time.

[0272] However, even if the system latency is the same 10 ms, if the prediction start time is 10 ms later than the timing when the latest RGB image was acquired, an optical flow is required that predicts an image 20 ms in the future, which is a system latency of 10 ms from the prediction start time, based on a reference RGB image acquired 10 ms before the prediction start time.

[0273] Furthermore, even if the system latency is the same 10 ms, if the prediction start time is 20 ms later than the timing when the latest RGB image was acquired, an optical flow is required that predicts an image 30 ms in the future, which is a system latency of 10 ms from the prediction start time, based on a reference RGB image acquired 20 ms before the prediction start time.

[0274] Similarly, even if the system latency is the same 10 ms, if the prediction start time is 30 ms later than the timing when the latest RGB image was acquired, an optical flow is required that predicts an image 40 ms in the future, which is a system latency of 10 ms from the prediction start time, based on a reference RGB image acquired 30 ms before the prediction start time.

[0275] This is the same whether the system latency is 20, 30, or 40 ms.

[0276] Therefore, in the present disclosure, expected optical flow variations are set for each system latency, a variation is selected based on the prediction start time and the reference time, which is the acquisition time of the RGB image that serves as the basis for warping, and an optical flow to be used is selected from the latest optical flow group based on the selected variation.

[0277] FIG. 30 shows the variations that can be selected for each of the system latencies of 10, 20, 30, and 40 (ms).

[0278] In Figure 30, each arrow in the figure represents a prediction result candidate OF-1 to OF-16, which is a selectable optical flow variation, and the end point of the arrow corresponds to the predicted time, and the start point of the arrow corresponds to the reference time when the reference RGB image was acquired.

[0279] Furthermore, time ts represents the prediction start time. Therefore, the prediction result candidates OF-1 to OF-4 each represent an optical flow predicted 10 ms into the future from the prediction start time ts.

[0280] The prediction result candidate OF-1 represents an optical flow that predicts an image that is 10 ms into the future, a system latency of 10 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts-30, which is 30 ms before the prediction start time ts.

[0281] The prediction result candidate OF-2 represents an optical flow that predicts an image that is 10 ms into the future, a system latency of 10 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts-20, which is 20 ms before the prediction start time ts.

[0282] Prediction result candidate OF-3 represents an optical flow that predicts an image that is 10 ms into the future, a system latency of 10 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts-10, which is 10 ms before the prediction start time ts.

[0283] Prediction result candidate OF-4 represents an optical flow that predicts an image that is 10 ms in the future, a system latency, from the prediction start time ts, by warping a reference RGB image acquired at the prediction start time ts.

[0284] The prediction result candidates OF-1 to OF-4 are all prediction result candidates that are 10 ms of system latency in the future from the prediction start time ts.

[0285] Hereinafter, prediction result candidates OF-1 to OF-4 with the same system latency of 10 ms will be referred to as a prediction result candidate group CA1.

[0286] In addition, prediction result candidate OF-5 represents an optical flow that predicts an image that is 20 ms in the future, a system latency of 20 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts-20, which is 20 ms before the prediction start time ts.

[0287] Prediction result candidate OF-6 represents an optical flow that predicts an image that is 20 ms into the future, a system latency of 20 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts-10, which is 10 ms before the prediction start time ts.

[0288] Prediction result candidate OF-7 represents an optical flow that predicts an image that is 20 ms in the future from the prediction start time ts, which is a system latency of 20 ms, by warping a reference RGB image acquired at the prediction start time ts.

[0289] Prediction result candidate OF-8 represents an optical flow that predicts an image that is 20 ms into the future, a system latency of 20 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts+10, which is 10 ms after the prediction start time ts.

[0290] The prediction result candidates OF-5 to OF-8 are all prediction result candidates that are 20 ms of system latency in the future from the prediction start time ts.

[0291] Hereinafter, prediction result candidates OF-5 to OF-8 with the same system latency of 20 ms will be referred to as a prediction result candidate group CA2.

[0292] Furthermore, prediction result candidate OF-9 represents an optical flow that predicts an image that is 30 ms into the future, a system latency of 30 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts-10, which is 10 ms before the prediction start time ts.

[0293] Prediction result candidate OF-10 represents an optical flow that predicts an image that is 30 ms in the future from the prediction start time ts, which is a system latency of 30 ms, by warping a reference RGB image acquired at the prediction start time ts.

[0294] Prediction result candidate OF-11 represents an optical flow that predicts an image that is 30 ms into the future, a system latency of 30 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts+10, which is 10 ms after the prediction start time ts.

[0295] Prediction result candidate OF-12 represents an optical flow that predicts an image that is 30 ms into the future, a system latency of 30 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts+20, which is 20 ms after the prediction start time ts.

[0296] The prediction result candidates OF-9 to OF-12 are all prediction result candidates that are 30 ms of system latency in the future from the prediction start time ts.

[0297] Hereinafter, prediction result candidates OF-9 to OF-12 with the same system latency of 30 ms will be referred to as a prediction result candidate group CA3.

[0298] Furthermore, prediction result candidate OF-13 represents an optical flow that predicts an image that is 40 ms in the future from the prediction start time ts, which is a system latency of 40 ms, by warping the reference RGB image acquired at the prediction start time ts.

[0299] Prediction result candidate OF-14 represents an optical flow that predicts an image that is 40 ms into the future, a system latency of 40 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts+10, which is 10 ms after the prediction start time ts.

[0300] Prediction result candidate OF-15 represents an optical flow that predicts an image that is 40 ms into the future, a system latency of 40 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts+20, which is 20 ms after the prediction start time ts.

[0301] Prediction result candidate OF-16 represents an optical flow that predicts an image that is 40 ms into the future, a system latency of 40 ms, from the prediction start time ts, by warping a reference RGB image acquired at time ts+30, which is 30 ms after the prediction start time ts.

[0302] The prediction result candidates OF-13 to OF-16 are all prediction result candidates that are 40 ms of system latency in the future from the prediction start time ts.

[0303] Hereinafter, prediction result candidates OF-13 to OF-16 with the same system latency of 40 ms will be referred to as a prediction result candidate group CA4.

[0304] <Optical Flow Selection Method> Next, a method for selecting an optical flow using prediction result candidate groups CA1 to CA4, which are variations of the optical flows described with reference to FIG. 30, will be described.

[0305] For example, consider the case where optical flow groups OFG21 to OFG24 exist in a time series as shown in Figure 31, and frames F1 to F5 of an RGB image are acquired every 30 ms from time t, i.e., at times t, t+30, t+60, t+90, and t+120.

[0306] Here, the optical flow group OFG21 is composed of optical flows OF31 to OF33 with prediction widths WP = 10, 20, and 30, respectively, from time t, represented by triangular marks at times t + 10, t + 20, and t + 30 arranged to the right in the figure, based on event data generated up to time t, represented by black circles.

[0307] Furthermore, the optical flow group OFG22 is composed of optical flows OF41 to OF43 with prediction widths WP = 10, 20, and 30, respectively, from time t+10, represented by triangular marks at times t+20, t+30, and t+40 arranged to the right in the figure, based on event data generated up to time t+10, represented by black circles.

[0308] Furthermore, the optical flow group OFG23 is composed of optical flows OF51 to OF53 with prediction widths WP = 10, 20, and 30, respectively, from time t+20, represented by triangular marks at times t+30, t+40, and t+50 arranged to the right in the figure, based on the event data generated up to time t+20, represented by black circles.

[0309] In addition, the optical flow group OFG24 is composed of optical flows OF61 to OF63 with prediction widths WP = 10, 20, and 30, respectively, from time t + 30, represented by triangular marks at times t + 40, t + 50, and t + 60, arranged to the right in the figure, based on event data generated up to time t + 30, represented by black circles.

[0310] 31, for example, it is assumed that the current time is time t and the latest optical flow group is optical flow group OFG21. In other words, it is assumed here that optical flow groups OFG22 to OFG24 do not exist.

[0311] In this case, the selection unit SL'' reads out the latest optical flow group OFG21 from the storage unit ME', and selects one of the optical flows OF31 to OF33 that make up the optical flow group OFG21.

[0312] On the other hand, with regard to the prediction result candidates, when the system latency is 10 ms, the prediction result candidates will be any of the variations of prediction result candidates OF-1 to OF-4 included in the prediction result candidate group CA1, which predicts a future image at time ts+10, 10 ms in the future from the prediction start time ts, out of the prediction result candidate group CA1 to CA4 shown in Figure 30.

[0313] 31 , the RGB image of frame F1, which serves as the reference for warping, is acquired at prediction start time t. Of the prediction result candidates OF-1 to OF-4 included in prediction result candidate group CA1, the variation whose prediction start time t matches the time when the RGB image of frame F1, which serves as the reference for warping, is acquired is prediction result candidate OF-4.

[0314] The selection unit SL'' selects the optical flow OF31, from among the optical flows OF31 to OF33 that constitute the read optical flow group OFG21, which coincides with the end point of the corresponding arrow when the prediction start time in the prediction result candidate OF-4 is matched with the reference time when the reference RGB image was acquired, as shown in Figure 31.

[0315] 31, for example, the current time is time t+10, and the latest optical flow group is optical flow group OFG22. In other words, it is assumed that this is a timing when optical flow groups OFG23 and OFG24 do not exist.

[0316] In this case, the selection unit SL'' reads out the latest optical flow group OFG22 from the storage unit ME', and selects one of the optical flows OF41 to OF43 that make up the optical flow group OFG22.

[0317] On the other hand, with regard to the prediction result candidates, when the system latency is 10 ms, the prediction result candidates will be any of the variations of prediction result candidates OF-1 to OF-4 included in the prediction result candidate group CA1, which predicts a future image at time ts+10, 10 ms in the future from the prediction start time ts, out of the prediction result candidate group CA1 to CA4 shown in Figure 30.

[0318] 31 , the RGB image of frame F1, which serves as the reference for warping, is acquired at time t, which does not coincide with the prediction start time t + 10. Of the prediction result candidates OF-1 to OF-4 included in prediction result candidate group CA1, the variation in which the reference time t, at which the RGB image of frame F1, which serves as the reference for warping, was acquired, is 10 ms later than the prediction start time t + 10, is prediction result candidate OF-3.

[0319] The selection unit SL'' selects the optical flow OF41, from among the optical flows OF41 to OF43 that constitute the read optical flow group OFG22, which coincides with the end point of the corresponding arrow when the prediction start time in the prediction result candidate OF-3 is matched with the reference time when the reference RGB image was acquired, as shown in Figure 31.

[0320] 31, for example, it is assumed that the current time is time t+20 and the latest optical flow group is optical flow group OFG23. In other words, it is assumed that this is a timing when optical flow group OFG24 does not exist.

[0321] In this case, the selection unit SL'' reads out the latest optical flow group OFG23 from the storage unit ME', and selects one of the optical flows OF51 to OF53 that make up the optical flow group OFG23.

[0322] On the other hand, with regard to the prediction result candidates, when the system latency is 10 ms, the prediction result candidates will be any of the variations of prediction result candidates OF-1 to OF-4 included in the prediction result candidate group CA1, which predicts a future image at time ts+10, 10 ms in the future from the prediction start time ts, out of the prediction result candidate group CA1 to CA4 shown in Figure 30.

[0323] 31 , the RGB image of frame F1, which serves as the reference for warping, is acquired at time t, which does not coincide with the prediction start time t + 20. Of the prediction result candidates OF-1 to OF-4 included in prediction result candidate group CA1, the variation in which the reference time t, at which the RGB image of frame F1, which serves as the reference for warping, was acquired, is 20 ms later than the prediction start time t + 20, is prediction result candidate OF-2.

[0324] The selection unit SL'' selects, from the read optical flows OF41 to OF43 constituting the optical flow group OFG22, the optical flow OF51 that coincides with the end point of the corresponding arrow when the prediction start time in the prediction result candidate OF-2 is matched with the reference time when the reference RGB image was acquired, as shown in Figure 31.

[0325] 31, for example, consider a case where the current time is time t+30, but the latest optical flow group OFG24 cannot be acquired due to a delay. That is, in this case, it is assumed that the latest optical flow group is optical flow group OFG23.

[0326] In this case as well, the selection unit SL'' will read out the latest optical flow group OFG23 from the storage unit ME', and one of the optical flows OF51 to OF53 that make up the optical flow group OFG23 will be selected.

[0327] On the other hand, when the system latency is 10 ms, the prediction result candidates are any of variations of prediction result candidates OF-1 to OF-4 included in prediction result candidate group CA1, which predicts a future image at time ts+10, which is 10 ms in the future from the prediction start time ts, among prediction result candidate groups CA1 to CA4 shown in Fig. 30. However, since it is time t+30 here, the prediction start time of optical flow group OFG23 is time t+20, while the current time is t+30, so in addition to the system latency of 10 ms, a delay of an additional 10 ms occurs.

[0328] Therefore, here, among the prediction result candidate groups CA1 to CA4, the prediction result candidate is one of variations of prediction result candidates OF-5 to OF-8 included in prediction result candidate group CA2, which predicts a future image at time ts+20, which is 20 ms in the future from the prediction start time ts.

[0329] 31 , the RGB image of frame F2, which serves as the reference for warping, was acquired at time t+30, which does not coincide with the prediction start time t+20 of optical flow group OFG23. Of the prediction result candidates OF-5 to OF-8 included in prediction result candidate group CA2, the variation in which the reference time is 10 ms ahead of the prediction start time, such as the reference time t+30 when the RGB image of frame F2, which serves as the reference for warping, was acquired and the prediction start time t+20, is prediction result candidate OF-8.

[0330] The selection unit SL'' selects the optical flow OF52, which coincides with the end point of the corresponding arrow when the prediction start time in the prediction result candidate OF-8 matches the reference time when the reference RGB image was acquired, from the read optical flows OF51 to OF53 that make up the optical flow group OFG23, as shown in Figure 31.

[0331] As described above, the relationship between the reference time and predicted start time of the optical flow, which can be set for each latency, as shown in Figure 30, is set as a variation called a prediction result candidate, and an appropriate optical flow is selected from the read optical flow group based on the relationship between the actual reference time and the predicted start time.

[0332] In other words, the optical flow variations are combinations of differential information between the predicted destination time, which is set for each latency and is based on the prediction start time, and the reference time, which is the time when the RGB image that serves as the basis for warping is acquired, that enable warping processing, i.e., enable the generation of a predicted image.

[0333] The selection unit SL'' estimates a variation from the relationship between the actual prediction start time set for each latency and the actual reference time among optical flows with multiple prediction widths corresponding to multiple latencies, and selects an appropriate optical flow from the optical flows with multiple prediction widths based on the estimated variation, thereby enabling prediction of future images by the system latency.

[0334] <Functions Realized by Information Processing Device of Third Embodiment> Next, functions realized by the information processing device 31''' of the third embodiment will be described with reference to the functional block diagram of Fig. 32. Note that in the information processing device 31''' of Fig. 32, components having the same functions as those in the information processing device 31'' of Fig. 18 or the information processing device 31 of Fig. 11 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0335] The functions realized by the information processing device 31 of the third embodiment include a data processing block 41a′ realized by the control unit 41, a display processing block 41b′, and a prediction result storage unit 151 realized by the storage unit 44.

[0336] The data processing block 41a' has a configuration corresponding to the data processing block 41a, and is composed of an EVS 49, a data acquisition unit 52, and a prediction unit 53'.The data processing block 41a' acquires event data generated by the EVS 49, predicts optical flows with multiple prediction ranges, and stores the predicted optical flows with multiple prediction ranges in the prediction result memory unit 151 as an optical flow group.

[0337] The display processing block 41b' accesses the prediction result storage unit 151, reads out the most recent optical flows, and sets a system delay value based on the read out optical flows.

[0338] Furthermore, the display processing block 41b' selects an optical flow based on the set system delay value and the timing of acquiring the RGB image.

[0339] Furthermore, the display processing block 41b' generates an RGB image of the future by the system latency by warping based on the selected optical flow, and controls the display device 161 to display the generated RGB image of the future by the system latency.

[0340] More specifically, the display processing block 41b' is made up of a system delay value setting unit 51'', a prediction result selection unit 54', and a prediction result utilization unit 55'.

[0341] The system delay value setting unit 51'' sets a system delay value in the same way as the system delay value setting units 51 and 51', but it accesses the prediction result storage unit 151 at a predetermined refresh rate, reads out the most recent optical flow group, sets a system delay value based on the time information (timestamp) in the optical flows that make up the optical flow group, and outputs the system delay value to the prediction result selection unit 54'.

[0342] The prediction result selection unit 54′ selects an optical flow that satisfies the system latency and the timing at which the RGB image is acquired from the most recent optical flows read out from the prediction result storage unit 151 based on the system delay value supplied from the system delay value setting unit 51″ and the timing at which the RGB image is acquired supplied from the data acquisition unit 52′, and outputs the selected optical flow to the prediction result utilization unit 55′.

[0343] The prediction result utilization unit 55' performs warping processing on the RGB image supplied from the data acquisition unit 52' based on the selected optical flow, generates a future predicted image for the system latency, and outputs it to the display device 161 for display.

[0344] The RGB camera 50 is made up of a CMOS image sensor or the like, and captures an RGB image with approximately the same angle of view as the EVS 49, and outputs it to a data acquisition unit 52'.

[0345] The data acquisition unit 52' has the same functions as the data acquisition unit 52, and outputs the RGB image supplied from the RGB camera 50 to the prediction result selection unit 54' and the prediction result utilization unit 55'.

[0346] With the above configuration, optical flows are predicted for each of a plurality of prediction widths based on the event data generated by the EVS 49, and the corresponding plurality of optical flows are stored in the prediction result storage unit 151 as an optical flow group.

[0347] In addition, the most recent optical flow group stored in the prediction result memory unit 151 at a predetermined refresh rate is read out, and a system delay value is set based on the time information (timestamp) of each optical flow that constitutes the read optical flow group.

[0348] Furthermore, based on the set system delay value and the acquisition timing of the RGB image captured by the RGB camera 50 supplied via the data acquisition unit 52', an optical flow predicted with a prediction width that takes into account the system latency and the timing at which the RGB image that serves as the basis for warping performed in the prediction result utilization unit 55' is acquired is selected from the read optical flows.

[0349] Then, based on the selected optical flow, the RGB image is warped to generate a predicted image for the future by the system latency, and the predicted image is displayed on the display device 181 .

[0350] As a result, a predicted image at a future time corresponding to the system latency is generated and the operation of the display device 181 is controlled, thereby making it possible to reduce the effect of latency.

[0351] The processing performed by the data processing block 41a' and the processing performed by the display processing block 41b' are performed in parallel but are not necessarily synchronized. In particular, the processing performed by the display processing block 41b' is performed at a refresh rate of, for example, 10 ms intervals in the display device 181.

[0352] <Information Processing by Data Processing Block in Information Processing Device of FIG. 32> Next, information processing by the data processing block 41a' of the information processing device 31''' of FIG. 32 will be described with reference to the flowchart of FIG.

[0353] In step S111 , the data acquisition unit 52 acquires event data, including the detection positions where events are detected, supplied from the EVS 49 for a predetermined period of time, and outputs the acquired data to the prediction unit 53 .

[0354] In step S112, the prediction unit 53 performs preprocessing for the prediction process by converting the event data for a predetermined time into frames of multiple channels for each prediction width, and thereby making it into a format that can be processed as a prediction model consisting of a DNN.

[0355] In step S113, the prediction unit 53' predicts optical flows for a plurality of prediction ranges from a plurality of most recent event data based on the preprocessed event data, and generates optical flows that are prediction results for each prediction range.

[0356] In step S114, the prediction unit 53' stores in the prediction result storage unit 151 a group of optical flows made up of a plurality of optical flows that are prediction results for each of a plurality of prediction widths.

[0357] In step S115, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S111, and the subsequent steps are repeated.

[0358] Then, in step S115, if an instruction to end the process is given, the process ends.

[0359] By the above processing, optical flow groups consisting of optical flows with multiple prediction widths are generated and sequentially stored repeatedly in the prediction result storage unit 151. Note that the optical flow groups stored in the prediction result storage unit 151 only need to store the most recent predetermined number of optical flow groups, so if the predetermined number is exceeded, the oldest ones may be gradually deleted and new optical flow groups may be stored.

[0360] <Information Processing by Display Processing Block in Information Processing Device of FIG. 32> Next, information processing by the display processing block 41b' of the information processing device 31''' of FIG. 32 will be described with reference to the flowchart of FIG.

[0361] In step S131, the data acquisition unit 52' acquires RGB data of the RGB image captured by the RGB camera 50, and outputs the data to the prediction result selection unit 54' and the prediction result utilization unit 55'.

[0362] In step S132, the prediction result selection unit 54′ reads and acquires the most recent optical flow group from the prediction result storage unit 151. At this time, the system delay value setting unit 51′″ acquires the optical flow group read directly from the prediction result storage unit 151 or indirectly from the prediction result selection unit 54.

[0363] In step S133, the system delay value setting unit 51''' calculates a system delay value based on the time information (time stamp) of the optical flow that is the starting point of the prediction that constitutes the optical flow group, and outputs it to the prediction result selection unit 54.

[0364] In step S134, the prediction result selection unit 54′ executes a prediction result selection process, and selects a corresponding optical flow from among the multiple optical flows that constitute the most recent optical flow group that has been read out, based on the set system delay value, i.e., the system latency, supplied from the system delay value setting unit 51′″ and the acquisition timing of the RGB image that serves as the basis for warping, and outputs the selected optical flow to the prediction result utilization unit 55′.

[0365] The prediction result selection process will be described in detail later with reference to the flowchart of FIG.

[0366] In step S135, the prediction result utilization unit 55′ performs warping processing based on the optical flow supplied from the prediction result selection unit 54′ and the RGB data of the RGB image to generate a predicted image for the future by the system latency, and outputs the image to the display device 181 for display.

[0367] In step S136, it is determined whether or not an instruction to end has been given, and if an instruction to end has not been given, the process proceeds to step S137.

[0368] In step S137, it is determined whether a predetermined time has elapsed, and the same processing is repeated until the predetermined time has elapsed. Once the predetermined time has elapsed, the processing returns to step S131, and the subsequent processing is repeated. In other words, the processing of steps S131 to S137 is repeated at predetermined time intervals unless an end command is given in step S136.

[0369] Then, in step S136, if an instruction to end the process is given, the process ends.

[0370] With the above configuration, optical flows are predicted for each of a plurality of prediction widths based on the event data generated by EVS 49, and the optical flows for the plurality of prediction widths that are the prediction results are stored in the prediction result memory unit 151 as an optical flow group.

[0371] In addition, a group of prediction result candidates is read out from the group of optical flows stored in the prediction result storage unit 151 according to the system latency, and based on the timing at which the RGB image is acquired, an optical flow corresponding to one of the group of prediction result candidates is selected as the optical flow that predicts a future image that is the system latency ahead.

[0372] Then, based on the selected optical flow, the RGB image captured by the RGB camera is warped to generate a predicted image for the future by the system latency, and is displayed on the display device 181.

[0373] As a result, an RGB image at a future time corresponding to the system latency is warped from the captured reference RGB image based on the selected optical flow, thereby reducing the impact of system latency on the display of RGB images captured by an RGB camera.

[0374] <Prediction Result Selection Process> Next, the prediction result selection process will be described with reference to the flowchart of FIG.

[0375] In step S151, the prediction result selection unit 54' selects one of the prediction result candidate groups CA1 to CA4 based on the system delay value, as described with reference to FIGS.

[0376] In step S152, as described with reference to Figures 30 and 31, the prediction result selection unit 54' identifies one of the prediction result candidates from the selected group of prediction result candidates based on the relationship between the current time, i.e., the prediction start time, and the reference time at which the RGB image was acquired, and selects a corresponding optical flow from the group of optical flows based on the identified prediction result candidate.

[0377] This makes it possible to select an optical flow that can appropriately generate a predicted image at a future time by warping, by the amount of system latency, based on the already acquired RGB image, even if the prediction start time is not the time when the RGB image is acquired.

[0378] As a result, it is possible to reduce the impact of latency that occurs when displaying images captured by an RGB camera installed with a similar angle of view to the EVS.

[0379] In the present disclosure, an example in which an RGB camera is used has been described, but a camera of another wavelength band may also be used, for example, a grayscale camera.

[0380] Furthermore, we have explained examples in which event data and optical flow with multiple prediction ranges are predicted based on event data obtained from an EVS, but other configurations may be used as long as they can detect changes in the amount of received light at a high frame rate and output data. For example, a SPAD (Single Photon Avalanche Diode) may be used instead of an EVS.

[0381] Although the above description deals with examples in which the system delay value is input by the user and examples in which it is dynamically measured and used, the user may be able to switch between either of these. Furthermore, when prediction results for multiple prediction ranges are selected based on the system delay value, the system may be able to switch between selecting a prediction result based on a system delay value input by the user and selecting a prediction result based on a dynamically measured and set system delay value.

[0382] Furthermore, in the first and second embodiments, we have described a prediction model that predicts the trajectory of a ball, but the prediction model may predict the movement of something other than a ball, and may, for example, predict the position of a human body or a vehicle body.

[0383] <<6. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0384] 36 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0385] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0386] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0387] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0388] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0389] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0390] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0391] 36 realizes the functions of the information processing device 31 in FIG. 11, the information processing device 31' in FIG. 14, the information processing device 31'' in FIG. 18, and the information processing device 31''' in FIG.

[0392] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0393] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0394] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0395] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0396] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0397] The present disclosure may also be configured as follows. <1> An information processing device comprising: a first data acquisition unit that acquires first data; and a prediction unit that processes the first data and outputs a plurality of prediction results, wherein the prediction unit sets a prediction range that is a candidate value for a system delay time and a plurality of times that include a time different from the latest acquisition time of the first data as reference times, and predicts difference information from each of the reference times at times that are the prediction range ahead, starting from the latest acquisition time of the first data. <2> The information processing device described in <1>, wherein the first data is event data detected by an EVS (Event-based Vision Sensor). <3> The information processing device described in <1>, further comprising: a prediction result selection unit that selects at least one prediction result from the plurality of prediction results; and a prediction result utilization unit that performs processing using the selected prediction result. <4> The information processing device according to <3>, further comprising a second data acquisition unit that acquires second data different from the first data, wherein the prediction result selection unit determines, based on an acquisition time of the second data, which prediction result corresponds to a reference time to select, and the prediction result utilization unit performs processing using the second data and the selected prediction result. <5> The information processing device according to <4>, wherein the second data is image data including at least luminance information. <6> The information processing device according to <4>, wherein the prediction unit predicts a motion vector of an object, and the prediction result utilization unit generates a prediction result for the second data based on the second data and the motion vector. <7> The information processing device according to <4>, further comprising a prediction result storage unit that stores the prediction result output by the prediction unit for a certain period of time, wherein the prediction result selection unit selects from the prediction results stored in the prediction result storage unit if the latest output result of the prediction unit has not been obtained.<8> The information processing device according to <7>, wherein the prediction result storage unit stores the result together with the acquisition time of the first data used to obtain the prediction result, and calculates the system delay time using the acquisition time of the first data stored in association with the prediction result stored in the prediction result storage unit as a starting point, and the prediction result selection unit selects the prediction result based on both the system delay time and the acquisition time of the second data. <9> The information processing device according to <1>, further comprising a plurality of prediction result selection units that select at least one prediction result from the plurality of prediction results, and further providing a function for a user to select which prediction result selection unit to use. <10> The information processing device according to <1>, wherein the prediction unit provides a plurality of prediction ranges that are candidate values ​​for the system delay time, and outputs a prediction result for a time ahead by each prediction range, starting from the latest acquisition time of the first data. <11> The information processing device according to <1>, wherein the system delay time is set by a user. <12> The information processing device according to <1>, wherein the system delay time is estimated based on an execution time of at least one of an imaging process and a prediction process. <13> The information processing device according to <12>, wherein the system delay time is dynamically estimated each time processing is performed. <14> The information processing device according to <3>, wherein the prediction result utilization unit draws an image based on the prediction result. <15> The information processing device according to <3>, wherein the prediction result utilization unit controls equipment based on the prediction result. <16> An information processing method comprising the steps of acquiring first data, processing the first data, and outputting a plurality of prediction results, wherein a prediction range that is a candidate value for the system delay time and a plurality of times including a time different from the latest acquisition time of the first data are set as reference times, and difference information from each of the reference times at times ahead by the prediction range is predicted, starting from the latest acquisition time of the first data.<17> A program that causes a computer to function as a first data acquisition unit that acquires first data, and a prediction unit that processes the first data and outputs a plurality of prediction results, wherein the prediction unit sets a prediction range that is a candidate value for a system delay time and a plurality of times that include a time different from the latest acquisition time of the first data as reference times, and predicts difference information from each of the reference times at a time that is the prediction range ahead, starting from the latest acquisition time of the first data.

[0398] REFERENCE SIGNS 31, 31', 31'', 31''' INFORMATION PROCESSING DEVICE, 41a, 41a' DATA PROCESSING BLOCK, 41b, 41b' DISPLAY PROCESSING BLOCK, 51, 51', 51'' SYSTEM DELAY VALUE SETTING UNIT, 52, 52' DATA ACQUISITION UNIT, 53, 53' PREDICTION UNIT, 54, 54' PREDICTION RESULT SELECTION UNIT, 55 PREDICTION RESULT UTILIZATION UNIT, 121 ARM ROBOT, 181 DISPLAY DEVICE

Claims

1. An information processing device comprising: a first data acquisition unit that acquires first data; and a prediction unit that processes the first data and outputs a plurality of prediction results, wherein the prediction unit sets a prediction range that is a candidate value for a system delay time and a plurality of times including a time different from the latest acquisition time of the first data as reference times, and predicts difference information from each of the reference times at a time that is the prediction range ahead, starting from the latest acquisition time of the first data.

2. The information processing device according to claim 1, wherein the first data is event data detected by an EVS (Event-based Vision Sensor).

3. The information processing device according to claim 1, further comprising: a prediction result selection unit that selects at least one prediction result from the plurality of prediction results; and a prediction result utilization unit that performs processing using the selected prediction result.

4. An information processing device as described in claim 3, further comprising a second data acquisition unit that acquires second data different from the first data, wherein the prediction result selection unit determines which prediction result corresponds to which reference time based on the acquisition time of the second data, and the prediction result utilization unit performs processing using the second data and the selected prediction result.

5. The information processing device according to claim 4, wherein the second data is image data including at least luminance information.

6. The information processing device according to claim 4, wherein the prediction unit predicts a motion vector of an object, and the prediction result utilization unit generates a prediction result of the second data based on the second data and the motion vector.

7. An information processing device as described in claim 4, further comprising a prediction result storage unit that stores the prediction result output by the prediction unit for a certain period of time, and wherein the prediction result selection unit selects from among the prediction results stored in the prediction result storage unit when the latest output result of the prediction unit is not available.

8. The information processing device of claim 7, wherein the prediction result storage unit stores the result together with the acquisition time of the first data used to obtain the prediction result, the system delay time is calculated based on the acquisition time of the first data stored in correspondence with the prediction result stored in the prediction result storage unit, and the prediction result selection unit selects a prediction result based on both the system delay time and the acquisition time of the second data.

9. An information processing device as described in claim 1, further comprising a plurality of prediction result selection units each for selecting at least one prediction result from the plurality of prediction results, and further providing a function for a user to select which prediction result selection unit to use.

10. The information processing device according to claim 1, wherein the prediction unit provides a plurality of prediction ranges which are candidate values ​​for the system delay time, and outputs prediction results for times that are the respective prediction ranges ahead, starting from the most recent acquisition time of the first data.

11. The information processing device according to claim 1, wherein the system delay time is set by a user.

12. The information processing device according to claim 1, wherein the system delay time is estimated based on the execution time of at least one of an imaging process and a prediction process.

13. The information processing device according to claim 12, wherein the system delay time is dynamically estimated each time a process is performed.

14. The information processing device according to claim 3, wherein the prediction result utilization unit renders an image based on the prediction result.

15. The information processing device according to claim 3, wherein the prediction result utilization unit controls a device based on the prediction result.

16. An information processing method comprising the steps of acquiring first data, processing the first data, and outputting a plurality of prediction results, wherein a prediction range which is a candidate value for system delay time and a plurality of times including a time different from the latest acquisition time of the first data are set as reference times, and difference information from each of the reference times at a time that is the prediction range ahead, starting from the latest acquisition time of the first data, is predicted.

17. A program for causing a computer to function as a first data acquisition unit that acquires first data, and a prediction unit that processes the first data and outputs a plurality of prediction results, wherein the prediction unit sets a prediction range that is a candidate value for a system delay time and a plurality of times including a time different from the latest acquisition time of the first data as reference times, and predicts difference information from each of the reference times at a time that is the prediction range ahead, starting from the latest acquisition time of the first data.

Images