Image processing device, system, image processing method, and device

Abstract

To provide a technique capable of predicting a future frame with high accuracy.SOLUTION: An image processing device comprises: first acquisition means for acquiring first frame data, which is frame data of a video of a subject, at a predetermined cycle; second acquisition means for acquiring event data, which is data that can be acquired at a cycle shorter than the predetermined cycle and is a detection result of a change in pixel value of the subject; detection means for detecting the movement of the subject at a cycle shorter than the predetermined cycle using the event data; and generation means for generating second frame data corresponding to a time later than the time corresponding to the first frame data from the first frame data and the movement of the subject.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, a system, an image processing method, and an apparatus. [Background technology]

[0002] There is a technology called Cross Reality (hereafter referred to as XR) that blends virtually created content with the real world. XR is a general term for virtual reality (hereafter referred to as VR), augmented reality (hereafter referred to as AR), mixed reality (hereafter referred to as MR), and substitutional reality (hereafter referred to as SR), and refers to all technologies that blend virtually created content and environments with real senses and spaces.

[0003] One device that uses XR is the head-mounted display (hereafter referred to as HMD). HMDs come in two types: optical see-through and video see-through. The optical see-through method uses optical systems such as prisms and half mirrors to overlay an electronic display image onto the scenery that can be seen through the lens. On the other hand, the video see-through method is a method that augments the real world by capturing images of the scenery in real time with a camera installed in front of the HMD, converting them into digital images, and synthesizing the images of the scenery with virtually created digital content.

[0004] The video see-through method has the advantage of providing a high level of integration between the real world and digital information, since digital content is synthesized onto images of the scenery that have been digitized. However, the video see-through method has the disadvantage that it can easily cause phenomena such as VR sickness and visually-induced motion sickness, and can make actions such as grabbing moving objects with the hands difficult. This is because with the video see-through method, a delay occurs between capturing the scenery and displaying it, which is equivalent to the time it takes to process the synthesis of the digital content onto the scenery image. The important thing when using the video see-through method is to display the scenery image on the display without delay, in line with the line of sight of the HMD wearer.

[0005] Patent Document 1 discloses a technique for reducing processing delays by calculating a motion vector between two frames and generating a predicted frame from the calculated motion vector. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2014-36357 A Summary of the Invention [Problem to be solved by the invention]

[0007] In the technology disclosed in Patent Document 1, a motion vector cannot be calculated until two frames are obtained. Also, the frequency of calculating the motion vector is the same as the frame rate. As a result, detailed movements of the subject between two frames cannot be considered, and future frames cannot be predicted with high accuracy. If imaging is performed at a high frame rate, the difference in imaging time between two frames becomes small, making it possible to consider detailed movements of the subject, but it requires high processing speed. In addition, the processing speed is insufficient only on a device such as an HMD, making it difficult to make the device stand-alone. Furthermore, since the amount of data of a frame is generally large, imaging at a high frame rate increases the amount of communication from an imaging element (image sensor) such as a CMOS that captures images to a processing device that processes the frames.

[0008] An object of the present invention is to provide a technique that can predict future frames with high accuracy. [Means for solving the problem]

[0009] The image processing device of the present invention is characterized by having a first acquisition means for acquiring first frame data, which is frame data of an image of a subject at a predetermined period, a second acquisition means for acquiring event data, which is data that can be acquired at a period shorter than the predetermined period and is the result of detecting changes in pixel values ​​of the subject, a detection means for detecting movement of the subject at a period shorter than the predetermined period using the event data, and a generation means for generating second frame data corresponding to a time later than the time to which the first frame data corresponds, from the first frame data and the movement of the subject.

[0010] The system of the present invention comprises an imaging device and a display device, wherein the imaging device comprises a first acquisition means for acquiring first frame data, which is frame data of an image of the subject, by imaging the subject at a predetermined period, a second acquisition means for acquiring event data, which is a result of detecting changes in pixel values ​​of the subject, and which is data that can be acquired at a period shorter than the predetermined period by detecting changes in pixel values ​​of the subject, a detection means for detecting movement of the subject at a period shorter than the predetermined period using the event data, and a transmission means for transmitting the first frame data and motion information regarding the movement of the subject to an outside of the imaging device, and the display device comprises a receiving means for receiving the first frame data and the motion information from an outside of the display device, a generation means for generating second frame data from the first frame data and the motion information, which corresponds to a time later than the time to which the first frame data corresponds, and a display means for displaying an image based on the second frame data.

[0011] The image processing method of the present invention is characterized by comprising the steps of: acquiring first frame data, which is frame data of an image of a subject at a predetermined period; acquiring event data, which is data that can be acquired at a period shorter than the predetermined period and is the result of detecting changes in pixel values ​​of the subject; detecting movement of the subject at a period shorter than the predetermined period using the event data; and generating, from the first frame data and the movement of the subject, second frame data corresponding to a time later than the time to which the first frame data corresponds.

[0012] The equipment of the present invention is an equipment having the above-mentioned image processing device, and is characterized in that it further has at least one of an optical device corresponding to the image processing device, a control device that controls the image processing device, a processing device that processes signals output from the image processing device, a display device that displays information obtained by the image processing device, a memory device that stores information obtained by the image processing device, and a mechanical device that operates based on the information obtained by the image processing device. Effect of the Invention

[0013] According to the present invention, future frames can be predicted with high accuracy. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an example of the configuration of an imaging device according to a first embodiment. [Diagram 2] 4 is a diagram illustrating an example of the configuration of an event data acquisition unit according to the first embodiment; FIG. [Diagram 3] 4 is a diagram illustrating an example of the configuration of a pixel of an event data acquisition unit according to the first embodiment. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a processing unit according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an outline of processing performed by a processing unit according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of the configuration of a processing unit according to the second embodiment. [Figure 7]FIG. 11 is a diagram showing an outline of processing by a processing unit according to a second embodiment. [Figure 8] FIG. 11 is a diagram illustrating an example of the configuration of a vision system according to a third embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of the configuration of a device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below can be arbitrarily combined as long as no contradiction occurs.

[0016] <Embodiment 1> [Overall description of the imaging device] The configuration of the imaging device according to the embodiment 1 will be described with reference to Fig. 1. The imaging device according to the embodiment 1 has an event data acquisition section 11, a frame data acquisition section 12, and a processing section 13.

[0017] The event data acquisition unit 11 detects a change in pixel values ​​of the subject, thereby acquiring event data that is a result of the detection of the change in pixel values. The frame data acquisition unit 12 captures an image of the subject, thereby acquiring frame data of the image of the subject. The processing unit 13 receives the event data from the event data acquisition unit 11, and detects the motion of the subject using the event data, thereby acquiring motion information that is a result of the detection of the motion of the subject. Furthermore, the processing unit 13 generates predicted frame data that corresponds to a time later than the time to which the frame data output by the frame data acquisition unit 12 corresponds, from the generated motion information and the frame data output by the frame data acquisition unit 12.

[0018] The frame data acquisition unit 12 captures images for acquiring frame data at a predetermined period (hereinafter, referred to as a frame period). The event data acquisition unit 11 can detect changes in pixel values ​​at a period shorter than the frame period, and can acquire event data at a period shorter than the frame period. Therefore, the processing unit 13 can detect the movement of the subject at a period shorter than the frame period using the event data, and can acquire movement information at a period shorter than the frame period. The processing unit 13 can predict future frames with high accuracy by using the movement information obtained by detecting the movement of the subject at a period shorter than the frame period, that is, with high time resolution. Specifically, the processing unit 13 can generate predicted frame data that reflects detailed movement of the subject between frames.

[0019] Each of the multiple images that make up a video (which may be interpreted as a moving image) is a frame. The period from when the frame synchronization signal becomes active once to when it becomes active again is one frame period, and one image generated during this one frame period is one frame. In general, an imaging element (image sensor) has multiple pixels arranged in a two-dimensional array (which may be interpreted as a matrix), a vertical scanning circuit that scans each row of the multiple pixels, and the like. The frame synchronization signal is a signal that controls the start timing of the vertical scanning circuit. Focusing on a certain pixel row, the period from when the signal of that pixel row is read out to when the signal of that pixel row is read out again is one frame period, and one image generated during that period is one frame. Frame data is the image data of a frame.

[0020] [Event data acquisition section] The circuit configuration of the event data acquisition unit 11 according to the first embodiment will be described with reference to Fig. 2. The event data acquisition unit 11 performs a predetermined calculation on a signal detected by a pixel to obtain event data. The event data acquisition unit 11 includes an event pixel unit 101 and an event readout unit 104.

[0021] The event pixel unit 101 has a plurality of pixels P(m,n) 103 arranged in a two-dimensional array, similar to a general image sensor. m is an integer satisfying 0≦m≦(M-1), n ​​is an integer satisfying 0≦n≦(N-1), and M×N is the number of pixels in the event data acquisition unit 11. The arrangement of the pixels 103 is not particularly limited and may be a one-dimensional line sensor-like arrangement or other arrangements.

[0022] Each of the pixels 103 can detect a change in the amount of incident light as an event and output an event signal E, which is the detection result, and is configured, for example, by a circuit shown in FIG. 3. The photodiode 210 generates a photocurrent Ip according to the amount of incident light. The logarithmic I / V converter 220 converts the photocurrent Ip into a potential and obtains a logarithmic potential Vl by performing a logarithmic function conversion. The subtractor 230 obtains a difference Vd between the logarithmic potential Vl and a reference potential. The comparator 240 compares the difference Vd with a predetermined threshold. The thresholds include a positive threshold T1 and a negative threshold T2, and an event is detected when the difference Vd exceeds the positive threshold T1 or falls below the negative threshold T2. The event signal E(x,y,t) of the pixel P(x,y) at time t can be expressed by the following formula.

number

[0023] The resolution of the time t is, for example, 1 μs. An event signal E=1 indicates that a change in the amount of incident light that becomes brighter has occurred, that is, the occurrence of a positive event, and an event signal E=-1 indicates that a change in the amount of incident light that becomes darker has occurred, that is, the occurrence of a negative event. The above formula is an example of the expression of the event signal E, and various expressions can be adopted, such as assigning different numerical values ​​to each state of a positive event, a negative event, and no event, or treating positive events and negative events as one type of event signal without distinguishing them. An event occurs with high time resolution, independently (asynchronously) of a frame synchronization signal used in a normal image sensor. The event signal E is sent to the event readout unit 104. The logarithmic potential Vl at the time of the occurrence of the event is used as the next reference potential. The reference potential is updated with the response signal Ack from the event readout unit 104 as a trigger.

[0024] The event pixel unit 101 outputs a transmission request signal Req of an event signal E to the event readout unit 104 for each pixel row 102, which is a row of multiple pixels 103. The transmission request signal Req from each pixel row 102 is output, for example, when at least one pixel 103 in the pixel row 102 detects an event, that is, when the event signal E becomes 1 or -1. Although the event pixel unit 101 has output a transmission request signal Req for each pixel row 102, the event pixel unit 101 may output a transmission request signal Req for each column of multiple pixels 103. The event pixel unit 101 may output a transmission request signal Req for each other region (pixel group), and may output a transmission request signal Req for each rectangular region consisting of a predetermined number of columns and a predetermined number of rows, for example.

[0025] The event readout unit 104 has an arbitration unit that arbitrates transmission request signals Req from each pixel row 102 of the event pixel unit 101, and reads out the event signal E of each pixel 103 in the selected pixel row 102. The event signal E held by the pixel 103 is cleared to 0 when it is read out by the event readout unit 104. Thereafter, a response signal Ack is returned from the event readout unit 104 to the pixel row 102 from which the event signal E was read out, and each pixel 103 in the pixel row 102 resumes event detection using the logarithmic potential Vl at the time when the response signal Ack was received as a new reference potential.

[0026] The event signal E read by the event readout unit 104 is used as final output data (event data) of the event data acquisition unit 11. For example, the event readout unit 104 outputs event data including the polarity of the event (positive or negative of the event), coordinate values ​​corresponding to the event, and a timestamp of the time when the event signal E was read out. The coordinate values ​​and the timestamp can be acquired by the event readout unit 104. For example, the event readout unit 104 acquires the row and column positions of the pixel 103 from which the event signal E was read out as coordinate values ​​corresponding to the event. In addition, the event readout unit 104 has, for example, a timer circuit (not shown), and acquires the output value of the timer at the time when the event signal E was read out as the timestamp.

[0027] The event data acquiring unit 11 may be a single-layered sensor or a multi-layered sensor. When the event data acquiring unit 11 is a multi-layered sensor, a boundary (Cu-Cu joint) between the first and second layers may be provided in the logarithmic I / V conversion unit 220.

[0028] [Explanation of frame data acquisition part] The following describes the configuration of the frame data acquisition unit 12. The frame data acquisition unit 12 is made up of an imaging element such as a CMOS, reads out charges accumulated in a photodiode, performs A / D conversion, and obtains frame data.

[0029] [Description of processing section] The configuration of the processing unit 13 according to the first embodiment will be described with reference to Fig. 4. The processing unit 13 includes a motion detection unit 131 and a frame data prediction unit 132.

[0030] The motion detection unit 131 acquires motion information, which is a detection result of the motion of the subject, using the event data output by the event data acquisition unit 11. The motion information indicates, for example, the type and amount of motion of the subject. The type of motion is x-direction translation, y-direction translation, yaw, pitch, roll, zoom, etc., and the amount of motion is the speed or acceleration of the motion.

[0031] The frame data prediction section 132 generates predicted frame data from the motion information output by the motion detection section 131 and the frame data output by the frame data acquisition section 12 .

[0032] The processing of the processing unit 13 will be described below with reference to Fig. 5. In Fig. 5, the right direction indicates the passage of time. Here, it is assumed that frame data is output from the frame data acquisition unit 12 at 60 fps, that is, at a cycle of approximately 16.67 msec, but the rate (frame rate) and cycle (frame cycle) at which the frame data acquisition unit 12 outputs the frame data are not particularly limited.

[0033] The motion detection unit 131 calculates an optical flow using the event data output by the event data acquisition unit 11 asynchronously with the output of frame data by the frame data acquisition unit 12. The optical flow is vector information that indicates the motion of each pixel or region, and is calculated based on the change in the occurrence position of an event that occurs in a predetermined region within a predetermined period. Furthermore, the motion detection unit 131 calculates the type and amount of motion from this optical flow. A known algorithm can be used for the method of calculating the optical flow from the event data and the method of calculating the type and amount of motion from the optical flow. Note that the method of calculating (obtaining) the type and amount of motion from the event data is not particularly limited, and for example, the type and amount of motion may be calculated directly from the event data without calculating the optical flow. Any method may be used to calculate the type and amount of motion from the event data. Here, it is assumed that the frame data acquisition unit 12 outputs frame data at time t. Then, the motion detection unit 131 calculates the type and amount of motion from the event data at time t+3, which is later than time t. Assume that the motion detection unit 131 acquires the type and amount of motion using the event data output by the event data acquisition unit 11, and generates motion information indicating the type and amount of motion. Specifically, the motion detection unit 131 acquires the type and amount of motion using the event data output by the event data acquisition unit 11 during the period from time t to time t+3, and generates motion information indicating the type and amount of motion. The motion detection unit 131 outputs the generated motion information to the frame data prediction unit 132. Note that optical flow may be used as the motion information.

[0034] The frame data prediction unit 132 generates predicted frame data using the motion information output by the motion detection unit 131 and the frame data output by the frame data acquisition unit 12. Here, it is assumed that the frame data prediction unit 132 generates frame data for time t+4 as predicted frame data using motion information generated from event data for the period from time t to time t+3 and the frame data for time t. At this point in time, the frame data for time t+4 is future frame data that has not yet been captured.

[0035] Since the motion information output by the motion detection unit 131 indicates the motion in the period from time t to time t3, the frame data prediction unit 132 predicts the motion in the period up to time t4 when generating (predicting) the frame data of time t+4. For example, the frame data prediction unit 132 generates (predicts) the motion information indicating the motion in the period from time t to time t+4 from the motion information indicating the motion in the period from time t to time t+3. The prediction method of the motion information is not particularly limited, but for example, using the ratio of the time (length of the period), it is possible to generate the motion information indicating the motion in which the motion from time t to time t+3 is extended by 4 / 3 times as the motion information indicating the motion in the period from time t to time t+4. Then, the frame data prediction unit 132 generates the predicted frame data of time t+4 using the motion information indicating the motion in the period from time t to time t+4 and the frame data of time t.

[0036] A known algorithm can be used as a method for predicting future frame data from frame data and motion information. Any method can be used as a method for predicting future frame data from frame data and motion information. When generating predicted frame data, additional processing such as filtering the boundary between moving pixels and non-moving pixels can be performed to suppress artifacts at the boundary between moving pixels and non-moving pixels.

[0037] Thereafter, in a similar manner, the motion detection unit 131 generates motion information from the event data for the period from time t+4 to time t+7. Then, the frame data prediction unit 132 generates predicted frame data for time t+8 using the motion information generated from the event data for the period from time t+4 to time t+7 and the predicted frame data for time t+4. This process is repeated until the frame data acquisition unit 12 outputs the frame data next to the frame data for time t. Here, it is assumed that the frame data acquisition unit 12 outputs the frame data next to the frame data for time t at time t+16.

[0038] The processing of the processing unit 13 when the frame data acquisition unit 12 outputs frame data at time t+16, i.e., the frame data next to the time t frame data, will be described. The motion detection unit 131 generates motion information from event data for the period from time t+16 to time t+19, and outputs the generated motion information to the frame data prediction unit 132. The frame data prediction unit 132 generates predicted frame data for time t+20 using the motion information generated from the event data for the period from time t+16 to time t+19 and the frame data output by the frame data acquisition unit 12 at time t+16.

[0039] As described above, according to the first embodiment, the use of event data makes it possible to detect the movement of a subject in a period shorter than the frame period. By using the motion of a subject detected at a period shorter than the period of the image, i.e., with high temporal resolution, future frames can be predicted with high accuracy.

[0040] Although an example has been described in which two sensors, the event data acquisition unit 11 and the frame data acquisition unit 12, a single hybrid sensor capable of acquiring both event data and frame data may be used.

[0041] Also, although an example has been described in which the image processing device that generates the predicted frame data is an imaging device, the image processing device that generates the predicted frame data does not have to be an imaging device. For example, an image processing device separate from the imaging device may obtain event data and frame data from the imaging device, detect the movement of a subject using the event data, and generate predicted frame data from the frame data and the movement of the subject.

[0042] <Embodiment 2> In the first embodiment, an example is described in which future frame data is predicted using frame data and motion information generated from event data. In the second embodiment, an example is described in which intermediate frame data is generated from predicted frame data generated by the method of the first embodiment and captured frame data (hereinafter referred to as captured frame data) in order to suppress artifacts when prediction is incorrect. Artifacts when prediction is incorrect include, for example, distortion of the shape of an object in a video, and a discrepancy between the movement of an object in a video and the actual movement. Although intermediate frame data is also a type of predicted frame data, hereinafter, only predicted frame data generated by the method of the first embodiment is referred to as predicted frame data, and the predicted frame data generated by the method of the first embodiment is distinguished from the intermediate frame data.

[0043] In the second embodiment, the processing unit 13 in Fig. 4 is not used, but the processing unit 23 in Fig. 6 is used. The processing unit 23 has a motion detection unit 131, a frame data prediction unit 132, and a frame data comparison unit 233. The processing of the motion detection unit 131 and the frame data prediction unit 132 in the processing unit 23 is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0044] The operation of the frame data comparison unit 233 will be described with reference to Fig. 7. The frame data comparison unit 233 compares the captured frame data output by the frame data acquisition unit 12 with the predicted frame data output by the frame data prediction unit 132, and determines whether or not the difference between the captured frame data and the predicted frame data is greater than a threshold value.

[0045] For example, the frame data comparison unit 233 compares the captured frame data with the predicted frame data corresponding to the time closest to the time corresponding to the captured frame data. Here, it is assumed that the frame data comparison unit 233 compares the captured frame data and predicted frame data corresponding to the same time. Specifically, the frame data comparison unit 233 compares the captured frame data at time t+16 with the predicted frame data at time t+16. In the comparison, the frame data comparison unit 233 calculates the difference between the value of the captured frame data and the value of the predicted frame data for each pixel, and calculates the integrated value of the differences for all pixels as the difference between the captured frame data and the predicted frame data.

[0046] When the integrated value is greater than the threshold, it means that the prediction of the frame data is incorrect. When the integrated value, which is the difference between the imaging frame data at time t+16 and the predicted frame data at time t+16, is greater than the threshold, the image does not transition smoothly between the predicted frame data at time t+16 and the predicted frame data at time t+20. Therefore, when the integrated value is greater than the threshold, the frame data comparator 233 generates and outputs intermediate frame data. When the integrated value, which is the difference between the imaging frame data at time t+16 and the predicted frame data at time t+16, is greater than the threshold, the image does not transition smoothly between the predicted frame data at time t+16 and the predicted frame data at time t+20. Intermediate frame data at time t+16 is generated from the predicted frame data at time t+16. If the integrated value is equal to or less than the threshold, the frame data comparison unit 233 outputs the predicted frame data generated by the frame data prediction unit 132. If the integrated value, which is the difference between the imaging frame data at time t+16 and the predicted frame data at time t+16, is equal to or less than the threshold, the predicted frame data at time t+16 is output.

[0047] The frame data comparison unit 233 generates, as intermediate frame data, frame data having a value (e.g., an average value) between the value of the captured frame data and the value of the predicted frame data for each pixel. The intermediate frame data may or may not have a value between the value of the captured frame data and the value of the predicted frame data over the entire area of ​​the subject (which may be read as an image). For example, only data corresponding to a partial area of ​​the subject (e.g., an area where the difference between the captured frame data and the predicted frame data is large) in the intermediate frame data may have a value between the value of the captured frame data and the value of the predicted frame data. In that case, data corresponding to the remaining area of ​​the subject in the intermediate frame data may have the same value as the value of the captured frame data, or may have the same value as the value of the predicted frame data. When obtaining data corresponding to a partial area of ​​the subject in the intermediate frame data (hereinafter referred to as a partial area), the data used in the captured frame data may or may not be data corresponding to an area in the same location as the partial area. Similarly, when obtaining data corresponding to a partial area in the intermediate frame data, the data used in the predicted frame data may or may not be data corresponding to an area in the same location as the partial area. The location of the area corresponding to the data used when obtaining data corresponding to a partial area of ​​the intermediate frame data may be different between the captured frame data and the predicted frame data.

[0048] When intermediate frame data is displayed, the display of the frame data is delayed by the time it takes to generate the intermediate frame data, compared to when predicted frame data is displayed. However, this delay is negligibly short and does not pose a major problem.

[0049] Furthermore, the processing by the frame data comparison unit 233 may be performed on all the imaging frame data output by the frame data acquisition unit 12, or may be performed only on some of the imaging frame data. For example, the processing may be performed once every few frames. Whether or not to perform the processing may be switched depending on the complexity of the imaging frame data. There are no particular limitations on the method of calculating the complexity of the imaging frame data, but for example, the complexity of the imaging frame data may be calculated by calculating an average pixel value of the imaging frame data, and based on the integrated value of the difference between each pixel value and the average value. The complexity of the imaging frame data may also be calculated based on the complexity of the movement detected from the event data.

[0050] As described above, according to the second embodiment, intermediate frame data is generated when the prediction of frame data is incorrect, so that a smooth image transition can be achieved even in such a case.

[0051] <Embodiment 3> In the third embodiment, an example of a system using an imaging device and a display device will be described. The configuration of a vision system according to the third embodiment will be described with reference to Fig. 8. The vision system according to the third embodiment includes an imaging device 31 and a display device 32.

[0052] The imaging device 31 has an event sensor unit 311, a frame sensor unit 312, and a transmission unit 313. The event sensor unit 311 has the functions of the event data acquisition unit 11 and the motion detection unit 131 in the first embodiment. The process of the frame sensor unit 312 is similar to the process of the frame data acquisition unit 12 in the first embodiment. The transmission unit 313 transmits an event The image capturing device 31 receives the motion information output by the motion sensor unit 311 and the frame data output by the frame sensor unit 312 and transmits them to the outside of the image capturing device 31.

[0053] The display device 32 has a receiving unit 321, a frame data prediction unit 322, and a display unit 323. The receiving unit 321 receives frame data and motion information output from the transmitting unit 313 of the imaging device 31 from outside the display device 32, and outputs the received frame data and motion information to the frame data prediction unit 322. The processing of the frame data prediction unit 322 is similar to the processing of the frame data prediction unit 132 in the first embodiment. The frame data prediction unit 322 generates predicted frame data using the motion information and frame data output by the receiving unit 321, and outputs the generated predicted frame data to the display unit 323. The display unit 323 displays an image based on the predicted frame data output from the frame data prediction unit 322.

[0054] As described above, according to the third embodiment, future frames can be predicted with high accuracy, as in the first embodiment. Furthermore, the imaging device 31 transmits motion information and frame data, and the display device 32 uses them to generate predicted frame data. This makes it possible to reduce the amount of communication between the imaging device 31 and the display device 32, compared to the case where the predicted frame data is sequentially transmitted from the imaging device 31 to the display device 32.

[0055] Although an example has been described in which the imaging device 31 generates the motion information, the device that generates the motion information does not have to be the imaging device 31. For example, event data may be transmitted from the imaging device 31 to the display device 32, and the motion information may be generated from the event data in the display device 32. Even in such a configuration, the amount of communication between the imaging device 31 and the display device 32 can be reduced compared to the case in which predicted frame data is sequentially transmitted from the imaging device 31 to the display device 32.

[0056] <Embodiment 4> Both of the first and second embodiments are applicable to the fourth embodiment. FIG. 9 is a schematic diagram for explaining a device 9191 including the semiconductor device 930 of this embodiment. The semiconductor device 930 can be any of the imaging devices (image processing devices) explained in the first and second embodiments, or an imaging device (image processing device) combining a plurality of embodiments. The device 9191 including the semiconductor device 930 will be explained in detail. As described above, the semiconductor device 930 can include a package 920 that houses the semiconductor device 910 in addition to the semiconductor device 910 having a semiconductor layer. The package 920 can include a base to which the semiconductor device 910 is fixed, and a cover such as glass that faces the semiconductor device 910. The package 920 can further include a bonding member such as a bonding wire or a bump that connects a terminal provided on the base and a terminal provided on the semiconductor device 910.

[0057] The device 9191 can include at least one of an optical device 940, a control device 950, a processing device 960, a display device 970, a storage device 980, and a mechanical device 990. The optical device 940 corresponds to the semiconductor device 930. The optical device 940 is, for example, a lens, a shutter, or a mirror. The control device 950 controls the semiconductor device 930. The control device 950 is, for example, a semiconductor device such as an ASIC.

[0058] The processing device 960 processes the signal output from the semiconductor device 930. The processing device 960 is a semiconductor device such as a CPU or ASIC for configuring an AFE (analog front end) or a DFE (digital front end). The display device 970 is an EL display device or a liquid crystal display device that displays information (images) obtained by the semiconductor device 930. The storage device 980 is a magnetic device or a semiconductor device that stores information (images) obtained by the semiconductor device 930. The storage device 980 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0059] The mechanical device 990 has a moving part or a propulsion part such as a motor or an engine. In the device 9191, the signal output from the semiconductor device 930 is displayed on the display device 970, or transmitted to the outside by a communication device (not shown) included in the device 9191. For this purpose, the device 9191 preferably further includes a memory device 980 and a processing device 960 in addition to the memory circuit and arithmetic circuit included in the semiconductor device 930. The mechanical device 990 may be controlled based on the signal output from the semiconductor device 930.

[0060] The device 9191 is also suitable for electronic devices such as information terminals (e.g., smartphones and wearable devices) with a photographing function and cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras). The mechanical device 990 in the camera can drive components of the optical device 940 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 990 in the camera can move the semiconductor device 930 for vibration isolation operation.

[0061] The device 9191 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 990 in the transportation equipment may be used as a moving device. The device 9191 as a transportation equipment is suitable for transporting the semiconductor device 930 or for assisting and / or automating driving (piloting) by using a photographing function. The processing device 960 for assisting and / or automating driving (piloting) can perform processing for operating the mechanical device 990 as a moving device based on information obtained by the semiconductor device 930. Alternatively, the device 9191 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0062] According to the above-described fourth embodiment, it is possible to obtain good pixel characteristics. Therefore, it is possible to increase the value of the semiconductor device 930. In this case, increasing the value corresponds to at least one of adding functions, improving performance, improving characteristics, improving reliability, improving manufacturing yield, reducing environmental load, reducing costs, making the device smaller, and reducing weight.

[0063] Therefore, if the semiconductor device 930 according to the fourth embodiment is used in the equipment 9191, the value of the equipment can be improved. For example, by mounting the semiconductor device 930 on a transport equipment, excellent performance can be obtained when photographing the outside of the transport equipment or measuring the external environment. Therefore, in manufacturing and selling the transport equipment, it is advantageous to decide to mount the semiconductor device 930 according to the fourth embodiment on the transport equipment in order to improve the performance of the transport equipment itself. In particular, the semiconductor device 930 is suitable for transport equipment that performs driving assistance and / or automatic driving of the transport equipment using information obtained by the semiconductor device 930.

[0064] The above-described embodiments may be modified as appropriate without departing from the technical concept. The disclosure of this specification includes not only what is described in this specification, but also all matters that can be understood from this specification and the drawings attached hereto. The disclosure of this specification also includes the complement of the concepts described in this specification. In other words, if this specification contains a statement that "A is greater than B," even if the statement that "A is not greater than B" is omitted, this specification still discloses that "A is not greater than B." This is because when it contains a statement that "A is greater than B," it is assumed that the case in which "A is not greater than B" is taken into consideration.

[0065] The disclosure of this embodiment includes the following configuration. (Configuration 1) a first acquisition means for acquiring first frame data, which is frame data of an image of a subject, at a predetermined period; The data can be acquired at a period shorter than the predetermined period, and the data indicates a change in pixel value of the object. A second acquisition means for acquiring event data which is a detection result of the occurrence of the event; a detection means for detecting a movement of the subject at a period shorter than the predetermined period by using the event data; a generating means for generating, from the first frame data and the movement of the subject, second frame data corresponding to a time later than the time corresponding to the first frame data; 13. An image processing device comprising: (Configuration 2) the first acquisition means captures an image of the subject at the predetermined period; The second acquisition means detects a change in the pixel value. 2. The image processing device according to claim 1, (Configuration 3) The generating means generates the second frame data based on the first frame data and the movement of the subject detected using the event data corresponding to a time later than the time corresponding to the first frame data. 3. The image processing device according to configuration 1 or 2. (Configuration 4) The image processing device further includes a second generating means for generating third frame data using third data having a value between first data corresponding to at least a partial region of the subject in the first frame data and second data corresponding to at least a partial region of the subject in the second frame data when a difference between the first frame data and the second frame data is greater than a threshold value. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) The value of the third data is an average value of the value of the first data and the value of the second data. 5. The image processing device according to configuration 4. (Configuration 6) The image processing device further includes a second generating means for generating third frame data using third data having a value between first data corresponding to a partial area of ​​a subject in the first frame data and second data corresponding to the partial area in the second frame data when a difference between the first frame data and the second frame data is greater than a threshold value. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 7) The value of the third data is an average value of the value of the first data and the value of the second data. 7. The image processing device according to configuration 6, (Configuration 8) The imaging device and the display device are included. The imaging device includes: a first acquisition means for acquiring first frame data, which is frame data of an image of a subject by capturing an image of the subject at a predetermined period; a second acquisition means for acquiring event data that is a result of the detection of the change in pixel value of the object, the event data being data that can be acquired at a period shorter than the predetermined period by detecting a change in pixel value of the object; a detection means for detecting a movement of the subject at a period shorter than the predetermined period by using the event data; a transmitting means for transmitting the first frame data and motion information relating to the motion of the subject to an outside of the imaging device; having The display device includes: a receiving means for receiving the first frame data and the motion information from outside the display device; a generating means for generating, from the first frame data and the motion information, second frame data corresponding to a time later than the time corresponding to the first frame data; a display means for displaying an image based on the second frame data; have A system characterized by: (Configuration 9) acquiring first frame data, which is frame data of an image of a subject, at a predetermined cycle; acquiring event data that can be acquired at a period shorter than the predetermined period and that is a detection result of a change in pixel value of the subject; detecting a movement of the subject at a period shorter than the predetermined period using the event data; generating, from the first frame data and the movement of the subject, second frame data corresponding to a time subsequent to the time corresponding to the first frame data; 13. An image processing method comprising: (Configuration 10) An apparatus having the image processing device according to any one of configurations 1 to 7, an optical device corresponding to the image processing device; A control device for controlling the image processing device; a processing device that processes a signal output from the image processing device; a display device for displaying information obtained by the image processing device; a storage device that stores information obtained by the image processing device; and and a mechanical device that operates based on information obtained by the image processing device. [Explanation of symbols]

[0066] 11: Event data acquisition section 12: Frame data acquisition section 13: Processing unit 131: Motion detection unit 132: Frame data prediction unit

Claims

1. A first acquisition means that acquires first frame data, which is frame data of the subject's video, at a predetermined interval, A second acquisition means for acquiring event data which is the result of detecting a change in the pixel value of the subject, and which can be acquired at a period shorter than the predetermined period. A detection means for detecting the movement of the subject at a period shorter than the predetermined period using the event data, A first generation means generates a second frame data corresponding to a time later than the time corresponding to the first frame data, from the first frame data and the movement of the subject, A second generation means generates a third frame data when the difference between the first frame data and the second frame data is greater than a threshold, using a third data having a value between a first data corresponding to at least a portion of the subject area in the first frame data and a second data corresponding to at least a portion of the subject area in the second frame data. An image processing apparatus characterized by having

2. A first acquisition means for acquiring first frame data, which is frame data of an image of a subject, at a predetermined period, A second acquisition means for acquiring event data which is the result of detecting a change in the pixel value of the subject, and which can be acquired at a period shorter than the predetermined period. A detection means for detecting the movement of the subject at a period shorter than the predetermined period using the event data, A first generation means generates a second frame data corresponding to a time later than the time corresponding to the first frame data, from the first frame data and the movement of the subject, When the difference between the first frame data and the second frame data is greater than a threshold, a second generation means generates third frame data using third data having a value between first data corresponding to a portion of the subject in the first frame data and second data corresponding to the portion of the second frame data. An image processing apparatus characterized by having

3. The first acquisition means captures the subject at the predetermined interval, The second acquisition means detects the change in the pixel value. The image processing apparatus according to feature 1.

4. The first generation means generates the second frame data based on the first frame data and the movement of the subject detected using the event data corresponding to a time later than the time corresponding to the first frame data. The image processing apparatus according to feature 1.

5. The value of the third data is the average of the value of the first data and the value of the second data. The image processing apparatus according to feature 1.

6. The value of the third data is the average of the value of the first data and the value of the second data. The image processing apparatus according to claim 2.

7. The first acquisition means acquires a plurality of first frame data corresponding to a plurality of time points, The first generation means generates a second frame data for each of the plurality of first frame data, which corresponds to a time later than the time to which the first frame data corresponds. The second generation means generates a sixth frame data using a sixth frame data having a value between a fourth frame data corresponding to at least a portion of the subject in the fourth frame data and a fifth frame data corresponding to at least a portion of the subject in the fifth frame data, when the difference between a fourth frame data, which is one of the plurality of first frame data, and a fifth frame data, which is the second frame data corresponding to the time closest to the time corresponding to the fourth frame data, is greater than the threshold. The image processing apparatus according to feature 1.

8. The first acquisition means acquires a plurality of first frame data corresponding to a plurality of time points, The first generation means generates a second frame data for each of the plurality of first frame data, which corresponds to a time later than the time to which the first frame data corresponds. The second generation means generates a sixth frame data using a sixth frame data having a value between the fourth data corresponding to a portion of the subject in the fourth frame data and the fifth data corresponding to the portion of the subject in the fifth frame data, when the difference between the fourth frame data, which is one of the plurality of first frame data, and the fifth frame data, which is the second frame data corresponding to the time closest to the time corresponding to the fourth frame data, is greater than the threshold. The image processing apparatus according to claim 2.

9. The output means further comprises outputting the second frame data when the difference is less than or equal to the threshold, and outputting the third frame data when the difference is greater than the threshold. The image processing apparatus according to feature 1.

10. The output means further comprises outputting the second frame data when the difference is less than or equal to the threshold, and outputting the third frame data when the difference is greater than the threshold. The image processing apparatus according to claim 2.

11. It has an imaging device and a display device, The imaging device is A first acquisition means that acquires first frame data, which is frame data of the image of the subject, by imaging the subject at a predetermined interval, A second acquisition means for acquiring event data, which is the result of detecting a change in the pixel value of the subject, and which can be acquired at a shorter period than the predetermined period, by detecting a change in the pixel value of the subject. A detection means for detecting the movement of the subject at a period shorter than the predetermined period using the event data, A transmission means for transmitting the first frame data and motion information relating to the movement of the subject to the outside of the imaging device. It has, The aforementioned display device is Receiving means for receiving the first frame data and the motion information from outside the display device, A first generation means generates a second frame data corresponding to a time later than the time corresponding to the first frame data from the first frame data and the motion information, A second generation means generates a third frame data using a third data having a value between a first data corresponding to at least a portion of the subject area in the first frame data and a second data corresponding to at least a portion of the subject area in the second frame data, when the difference between the first frame data and the second frame data is greater than a threshold, Display means for displaying an image based on the second frame data or the third frame data has A system characterized by the following features.

12. Having an imaging device and a display device, The imaging device is A first acquisition means that acquires first frame data, which is frame data of the image of the subject, by imaging the subject at a predetermined interval, A second acquisition means for acquiring event data, which is the result of detecting a change in the pixel value of the subject, and which can be acquired at a shorter period than the predetermined period, by detecting a change in the pixel value of the subject. A detection means for detecting the movement of the subject at a period shorter than the predetermined period using the event data, A transmission means for transmitting the first frame data and motion information relating to the movement of the subject to the outside of the imaging device. It has, The aforementioned display device is Receiving means for receiving the first frame data and the motion information from outside the display device, A first generation means generates a second frame data corresponding to a time later than the time corresponding to the first frame data from the first frame data and the motion information, A second generation means generates a third frame data using a third data having a value between a first data corresponding to a portion of the subject in the first frame data and a second data corresponding to the portion of the second frame data, when the difference between the first frame data and the second frame data is greater than a threshold, Display means for displaying an image based on the second frame data or the third frame data has A system characterized by the following features.

13. The receiving means receives a plurality of first frame data corresponding to a plurality of time points, The first generation means generates a second frame data for each of the plurality of first frame data, which corresponds to a time later than the time to which the first frame data corresponds. The second generation means generates a sixth frame data using a sixth frame data having a value between a fourth frame data corresponding to at least a portion of the subject in the fourth frame data and a fifth frame data corresponding to at least a portion of the subject in the fifth frame data, when the difference between a fourth frame data, which is one of the plurality of first frame data, and a fifth frame data, which is the second frame data corresponding to the time closest to the time corresponding to the fourth frame data, is greater than the threshold. The system according to feature 11.

14. The receiving means receives a plurality of first frame data corresponding to a plurality of time points, The first generation means generates a second frame data for each of the plurality of first frame data, which corresponds to a time later than the time to which the first frame data corresponds. The second generation means generates a sixth frame data using a sixth frame data having a value between the fourth data corresponding to a portion of the subject in the fourth frame data and the fifth data corresponding to the portion of the subject in the fifth frame data, when the difference between the fourth frame data, which is one of the plurality of first frame data, and the fifth frame data, which is the second frame data corresponding to the time closest to the time corresponding to the fourth frame data, is greater than the threshold. The system according to feature 12.

15. The display means displays the video based on the second frame data if the difference is less than or equal to the threshold, and displays the video based on the third frame data if the difference is greater than the threshold. The system according to feature 11.

16. The display means displays the video based on the second frame data if the difference is less than or equal to the threshold, and displays the video based on the third frame data if the difference is greater than the threshold. The system according to feature 12.

17. A process of acquiring first frame data, which is frame data of the subject's video, at a predetermined interval, A step of acquiring event data which is data that can be acquired at a shorter period than the predetermined period, and which is the result of detecting a change in the pixel value of the subject, A step of detecting the movement of the subject at a period shorter than the predetermined period using the event data, A step of generating a second frame data corresponding to a time later than the time corresponding to the first frame data from the first frame data and the movement of the subject, If the difference between the first frame data and the second frame data is greater than a threshold, a third frame data is generated using third data having a value between first data corresponding to at least a portion of the subject area in the first frame data and second data corresponding to at least a portion of the subject area in the second frame data. An image processing method characterized by having the following features.

18. A step of acquiring first frame data, which is frame data of the image of the subject, at a predetermined period, A step of acquiring event data which is data that can be acquired at a shorter period than the predetermined period, and which is the result of detecting a change in the pixel value of the subject, A step of detecting the movement of the subject at a period shorter than the predetermined period using the event data, A step of generating a second frame data corresponding to a time later than the time corresponding to the first frame data from the first frame data and the movement of the subject, If the difference between the first frame data and the second frame data is greater than a threshold, a third frame data is generated using third data having a value between first data corresponding to a portion of the subject in the first frame data and second data corresponding to the portion of the second frame data. An image processing method characterized by having the following features.

19. Multiple first frame data corresponding to multiple time points are acquired, For each of the plurality of first frame data, a second frame data is generated that corresponds to a time later than the time corresponding to the first frame data. If the difference between the fourth frame data, which is one of the plurality of first frame data, and the fifth frame data, which is the second frame data corresponding to the time closest to the time corresponding to the fourth frame data, is greater than the threshold, then the sixth frame data is generated using sixth frame data having a value between the fourth data corresponding to at least a portion of the subject area in the fourth frame data and the fifth data corresponding to at least a portion of the subject area in the fifth frame data. The image processing method according to feature 17.

20. Multiple first frame data corresponding to multiple time points are acquired, For each of the plurality of first frame data, a second frame data is generated that corresponds to a time later than the time corresponding to the first frame data. If the difference between the fourth frame data, which is one of the plurality of first frame data, and the fifth frame data, which is the second frame data corresponding to the time closest to the time corresponding to the fourth frame data, is greater than the threshold, then the sixth frame data is generated using sixth frame data having a value between the fourth data corresponding to a part of the subject in the fourth frame data and the fifth data corresponding to the part of the subject in the fifth frame data. The image processing method according to feature 18.

21. The further step of outputting the second frame data if the difference is less than or equal to the threshold, and outputting the third frame data if the difference is greater than the threshold. The image processing method according to feature 17.

22. The further step of outputting the second frame data if the difference is less than or equal to the threshold, and outputting the third frame data if the difference is greater than the threshold. The image processing method according to feature 18.

23. A device having an image processing apparatus according to any one of claims 1 to 10, Optical device corresponding to the aforementioned image processing device, A control device for controlling the aforementioned image processing device, A processing unit that processes the signal output from the aforementioned image processing unit, A display device that displays the information obtained by the image processing device, A storage device for storing information obtained by the image processing device, and The apparatus is characterized by further comprising at least one of the following: a mechanical device that operates based on information obtained by the image processing device.

Images