Event based vision sensor, image processing device, and image processing method

Abstract

An event based vision sensor according to the present disclosure comprises: an event detection unit that detects the occurrence of an event that is a luminance change of light received by a photodiode; an additional information generation unit that generates additional information including event data indicating the content of the detected event and a calculation method for an event threshold used when detecting the event; and a data output unit that outputs the additional information.

Description

Image Sensor, Image Processing Apparatus, and Image Processing Method

[0001] The present disclosure relates to an image sensor, an image processing apparatus, and an image processing method.

[0002] In the shooting of an object, an image sensor (hereinafter referred to as an EVS (Event based Vision Sensor)) that detects luminance changes for each pixel in real time as an event is used.

[0003] As a technology related to EVS, a technology for enhancing the versatility of EVS by including information such as an event threshold value and the presence or absence of flicker in the data output by EVS is known (for example, Patent Document 1).

[0004] In addition, as image processing using EVS, 3D rendering using a neural network is also actively used. For example, it is known to apply EVS to a technique called NeRF (Neural Radiance Fields) that uses a plurality of photos taken of an object from various angles as learning data, learns the representation of a scene including the object in space, and generates an image from a new viewpoint (for example, Non-Patent Document 1).

[0005] International Publication No. 2023 / 058669

[0006] “E-NeRF: Neural Radiance Fields from a Moving Event Camera” Simon Klenk, Lukas Koestler, Davide Scaramuzza, Daniel Cremers

[0007] According to the prior art, in processing using NeRF, even if the number of captured RGB images is small, it is possible to reconstruct an image with high accuracy.

[0008] By the way, when NeRF is executed using an EVS, in principle, learning is performed on the premise that the event threshold set in the EVS provided in the imaging device (hereinafter referred to as the "actual device") used for imaging matches the event threshold applied in the NeRF space. At this time, if the event threshold in the actual device matches the event threshold applied in the NeRF space, there is no problem. However, since there are various devices used for imaging, the event thresholds of both do not necessarily match. If the event thresholds of both do not match, the learning accuracy of the neural network in NeRF may decrease.

[0009] Therefore, the present disclosure proposes an image sensor, an image processing apparatus, and an image processing method capable of improving the learning accuracy of a neural network.

[0010] In order to solve the above problems, an image sensor according to one aspect of the present disclosure includes an event detection unit that detects the occurrence of an event, which is a change in the luminance of light received by a photodiode, an additional information generation unit that generates additional information including a calculation method of an event threshold used when detecting the event, together with event data indicating the content of the detected event, and a data output unit that outputs the additional information.

[0011] A diagram showing an overview of the image processing system according to the embodiment. A diagram showing an overview of NeRF. A diagram showing the information processing of NeRF. A diagram showing the information processing of NeRF when using event data. A diagram for explaining the detection of event data in the real space. A diagram for explaining the detection of event data in the NeRF space. A diagram for explaining the determination of event error. A diagram showing the information output by the EVS according to the present disclosure. A diagram showing a specific example of error calculation according to the present disclosure. A diagram showing a configuration example of the image processing apparatus according to the embodiment. A diagram showing a first configuration example of the control unit. A flowchart showing the processing flow in the first configuration example of the control unit. A flowchart related to the camera position estimation processing. A flowchart related to the event ray calculation processing. A flowchart related to the error calculation. A diagram for explaining the error calculation according to the embodiment. A diagram showing an overview of the image processing in the second configuration example of the control unit. A diagram showing the second configuration example of the control unit according to the embodiment. A diagram showing an overview of the image processing in the third configuration example of the control unit. A diagram for explaining the occurrence of an event. A diagram for explaining the output of pixel position information. A diagram showing the third configuration example of the control unit according to the embodiment. A flowchart showing the processing flow in the third configuration example of the control unit. A flowchart related to the calculation of non-event rays. A flowchart related to the non-event error calculation. A diagram (1) showing an example of the data structure output by the EVS. A diagram (2) showing an example of the data structure output by the EVS. A hardware configuration diagram showing an example of a computer that realizes the functions of the image processing apparatus.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant explanations are omitted.

[0013] The present disclosure will be described in the following order of items: 1. Embodiment 1-1. Overview of the image processing system according to the embodiment 1-2. Overview of the image processing according to the embodiment 1-3. Configuration of the image processing apparatus according to the embodiment 1-4. First configuration example of the control unit 1-5. Second configuration example of the control unit 1-6. Third configuration example of the control unit 1-7. Data structure 1-8. Modification example according to the embodiment 2. Other embodiments 3. Effects of the image sensor and the image processing apparatus according to the present disclosure 4. Hardware configuration

[0014] (1. Embodiment) (1-1. Overview of the image processing system according to the embodiment) First, the overview of the image processing system 1 according to the embodiment will be described using FIG. 1. FIG. 1 is a diagram showing the overview of the image processing system 1 according to the embodiment.

[0015] In FIG. 1, the image processing system 1 includes an EVS 10 and an image processing apparatus 100. The EVS 10 and the image processing apparatus 100 are connected via, for example, a data bus 14.

[0016] The EVS 10 is an image sensor that detects luminance changes for each pixel in real time as events, and transmits event data indicating the content of the events to the image processing apparatus 100 via the data bus 14. The EVS 10 includes a luminance detection unit 21, an event detection unit 22, an additional information generation unit 23, and a data output unit 24.

[0017] For example, the EVS 10 may have a stacked structure in which two chips, a pixel chip 25 provided with the luminance detection unit 21 and a signal processing chip 26 provided with the event detection unit 22, the additional information generation unit 23, and the data output unit 24, are stacked. The event detection unit 22 is, for example, an analog circuit that serves as an AFE (Analog Front End). Therefore, the EVS 10 may have a stacked structure in which three chips, a pixel chip provided with the luminance detection unit 21, an AFE provided with the event detection unit 22, and a logic chip provided with the additional information generation unit 23 and the data output unit 24, are stacked.

[0018] The data bus 14 transmits and receives data between the EVS 10 and the image processing device 100 in accordance with CSI-2 (Camera Serial Interface-2), which is an interface standard by, for example, the MIPI (Mobile Industry Processor Interface) Alliance.

[0019] The luminance detection unit 21 is configured to have a photodiode provided for each pixel, detects the luminance of the light received by the photodiode, and supplies a luminance signal indicating the luminance value to the event detection unit 22.

[0020] The event detection unit 22 calculates the difference between the luminance value indicated by the luminance signal supplied from the luminance detection unit 21 and a predetermined reference value in time series, and detects the occurrence of an event when the difference exceeds the positive-side event detection threshold or the negative-side event detection threshold. When the event detection unit 22 detects the occurrence of an event, it outputs event data indicating the content of the event (for example, data indicating whether the luminance value has changed to the positive side (positive) or the negative side (negative) from the reference value).

[0021] The additional information generation unit 23 generates various additional information provided additionally for the event data based on the event data output from the event detection unit 22, and supplies it to the data output unit 24. For example, the additional information generation unit 23 can generate various information regarding the event data as described later in addition to the embedded data defined by CSI-2 as the additional information.

[0022] The data output unit 24 outputs the event data output from the event detection unit 22 and the additional information supplied from the additional information generation unit 23 to the image processing device 100 in a frame configuration compliant with the standard of the data bus 14.

[0023] The image processing device 100 is an information processing device such as a server device or a PC (Personal Computer), and executes image processing according to the embodiment. For example, the image processing device 100 performs various data processes on the event data transmitted from the EVS 10 and acquires various information related to the event.

[0024] Further, the image processing device 100 executes image processing for obtaining an arbitrary viewpoint image of an object using a technique called NeRF. Specifically, the image processing device 100 uses various information obtained from the EVS in NeRF to perform learning of a neural network with higher accuracy compared to the case where such information is not used, and improves the accuracy of arbitrary viewpoint image generation.

[0025] (1-2. Outline of Image Processing According to the Embodiment) Next, the outline of NeRF, which is a premise for explaining the technology of the present disclosure, will be described using FIG. 2.

[0026] FIG. 2 is a diagram showing an outline of NeRF. NeRF is a technique that uses a plurality of photos taken of an object from various angles as learning data, learns the representation in the space of a scene including the object, and generates an image from a new viewpoint. That is, the user can obtain a free viewpoint image of the object to which NeRF is applied.

[0027] In the example of FIG. 2, it is assumed that the user obtains a free viewpoint image of the object 50 based on a plurality of images obtained by photographing the object 50 from several angles. The image 51 shown in FIG. 2 is an image obtained by photographing the left side of the object 50 using the EVS 10 (in this example, the EVS 10 may be synonymous with a camera). Similarly, the image 52 is an image obtained by photographing the object 50 from the front left diagonal. Similarly, the image 53 is an image obtained by photographing the object 50 from the front. Similarly, the image 54 is an image obtained by photographing the right side of the object 50.

[0028] The image processing device 100 acquires the plurality of images 51 to 54 etc. photographed by the user, applies NeRF to the images 51 to 54 (step S01), and obtains a free viewpoint image 60 of the object 50.

[0029] Next, information processing in NeRF will be described using FIG. 3. FIG. 3 is a diagram showing the information processing of NeRF.

[0030] Image generation (image rendering) in NeRF is performed by inferring what color a certain coordinate on a certain ray has in the NeRF space 61. Specifically, in NeRF, when the NeRF space 61 is viewed from a certain angle by a virtual camera, the object is reconstructed in the NeRF space 61 based on the probability of what color exists at which position (coordinate) on that ray.

[0031] In the example of FIG. 3, the position on the ray is indicated by the coordinate data 62. The coordinate data 62 includes spatial coordinate values that specify a position in the NeRF space and an angle that specifies the ray (in other words, the position of the virtual camera).

[0032] When the coordinate data 62 is input into the neural network (NN) 63, it is output as color data 64 that includes the RGB value and color density (σ) at that coordinate.

[0033] The image processing apparatus 100 performs rendering 65 based on the color data of a plurality of points on the obtained ray (that is, sampled data). That is, the image processing apparatus 100 can render the modeled three-dimensional space (NeRF space) as an image captured by a virtual camera installed at a certain position.

[0034] Subsequently, the image processing apparatus 100 compares the obtained inference value 66 (rendered image) with the correct data 68, which is an image actually taken of the object 50. The image processing apparatus 100 treats the difference between the inference value 66 and the correct data 68 as a loss, and learns the neural network 63 so as to reduce that loss. That is, the image processing apparatus 100 learns the difference in RGB values of how the actual object 50 appears (is rendered) in the NeRF space 67. Thereby, the image processing apparatus 100 can obtain a learned neural network for applying NeRF to the object 50.

[0035] In the example of FIG. 3, an example of performing learning related to NeRF based on a plurality of RGB images was shown. Further, when the EVS 10 can acquire event data together with the RGB images, the image processing apparatus 100 can perform learning related to NeRF using the event data. Such an example will be described using FIG. 4. FIG. 4 is a diagram showing the information processing of NeRF when event data is used.

[0036] In the example of FIG. 4, similar to FIG. 2, the user obtains a free viewpoint image of the object 50 based on a plurality of images obtained by photographing the object 50 from several angles. At this time, the EVS 10 acquires event data (positive events indicating that the luminance has risen above the threshold value, and negative events indicating that the luminance has fallen below the threshold value) together with the images.

[0037] The image processing apparatus 100 applies NeRF with event data added, that is, E-NeRF, based on the RGB data shown in the images 51 to 54 and the event data (step S02). Thereby, the image processing apparatus 100 acquires an arbitrary viewpoint image 70 regarding the object 50.

[0038] When performing learning of NeRF using event data, it is necessary to detect an error in the event data in addition to the error in the RGB values. The error detection when using event data will be described using FIGS. 5 and 6. FIG. 5 is a diagram for explaining the detection of event data in the real space.

[0039] In the example shown in FIG. 5, at a certain time t ｋ－１ and time t ｋ shows a process of detecting whether an event has occurred when photographing the object 50 between them. Specifically, in FIG. 5, the EVS 10 explains a process of detecting whether an event has occurred at a certain pixel u ｋ when photographing the object 50.

[0040] In the example of FIG. 5, whether an event has occurred at the pixel u ｋ is determined by comparing the time t ｋ－１ and the time t ｋIt is determined whether the difference in luminance (e.g., logarithmic difference) between each and a certain value exceeds a preset event detection setting value. Specifically, in Equation 71, for a certain pixel u, the EVS 10 ｋ determines that a negative or positive event has occurred when the luminance difference at a certain pixel u reaches a predetermined event threshold value (in the example of FIG. 5, "C"), and records the generated event data. More specifically, for the image obtained at time point t ｋ－１ and the image obtained at time point t ｋ they are recorded as a pair where event data has occurred.

[0041] In order to use the occurrence of an event in the real space for the learning of NeRF, it is assumed that an event has occurred in the NeRF space as well. This will be described using FIG. 6. FIG. 6 is a diagram for explaining the detection of event data in the NeRF space.

[0042] Whether an event has occurred in the NeRF space 72 is determined by rendering an image based on the ray passing through the NeRF space 72, and whether the luminance difference value at a certain pixel u ｋ exceeds the event threshold value. For the pixels corresponding to the pair where an event has occurred in FIG. 5, the premise that an event has occurred in the NeRF space 72 also holds. That is, the image processing apparatus 100 assumes that an event has occurred in the rendering results of the rays corresponding to the camera positions at time point t ｋ－１ and time point t ｋ which is a pair where an event has occurred in the real space data input to the NeRF space 72.

[0043] In the example of FIG. 6, pixel 731 in image 73 indicates one pixel rendered at the camera position at time point t ｋ－１ . Also, pixel 741 in image 74 indicates one pixel rendered at the camera position at time point t ｋ . The image processing apparatus 100 obtains the luminance difference between the pixels forming this pair. Specifically, the image processing apparatus 100 obtains the logarithmic difference in luminance between the two rendering results, as shown in Equation 75.

[0044] Then, the image processing apparatus 100 determines an event error based on the luminance difference value of the pixels obtained in the real space and the NeRF space. This point will be described using FIG. 7. FIG. 7 is a diagram for explaining the determination of the event error.

[0045] As shown in FIG. 7, when an event is detected in the real space, the luminance difference thereof is a positive or negative event threshold (p k C). In the NeRF space based on such input data, since the same event should occur, the difference value of the pixels corresponding to the pair of events occurring in the real space should match the event threshold. That is, as shown in Equation 76, the event error is the event threshold and the luminance difference value in the NeRF space, and the image processing apparatus 100 may train the neural network so that such a value is minimized. Specifically, Equation 76 is an equation for comparing the degree of coincidence between the logarithmic luminance difference of the rendering result in the NeRF space and the event threshold. The method of minimizing such an event error is, that is, the learning process in NeRF when event data is used in addition to the RGB image.

[0046] As described above up to FIG. 7, in the learning process using event data, an example in which the logarithmic difference of the luminance value is used has been shown. However, since the actual machines used in NeRF are diverse, event data is not always acquired using the same threshold or calculation method. Therefore, when performing the learning process in NeRF, it is necessary to acquire the processing content of how the actual machine obtains the event data. Specifically, in the learning process, it is necessary that the event generation logic of the actual machine and the logic for calculating the difference between the rendering results at two times that form a pair in the NeRF space are consistent.

[0047] For example, in an actual device, the threshold for detecting a positive event (hereinafter referred to as the "positive threshold" or "bright transition threshold") and the threshold for detecting a negative event (hereinafter referred to as the "negative threshold" or "dark transition threshold") may be different. Also, there may be differences in the method of determining the event threshold in the actual device, such as whether it uses a logarithm as shown up to FIG. 7, or whether it uses a linear function. In addition, in the detection of event data, there is also an EVS that performs predetermined filtering or skipping processing to suppress the frequent occurrence of event data.

[0048] If the information of these actual devices cannot be obtained accurately, the event error shown in FIG. 7 cannot be calculated accurately, so appropriate learning cannot be performed with NeRF. Therefore, the EVS 10 according to the present disclosure solves the above problems by outputting the information necessary for learning. This point will be described using FIG. 8. FIG. 8 is a diagram showing the information output by the EVS 10 according to the present disclosure.

[0049] As shown in FIG. 8, the EVS 10 outputs the actual device information 80 together with the RGB image and event data. The actual device information 80 is, for example, the calculation method of the luminance value for detecting the occurrence of an event, the event threshold, etc., preset in the EVS 10. The image processing device 100 acquires the actual device information 80 output from the EVS 10 and performs error calculation based on the acquired information. Thereby, the image processing device 100 can perform highly accurate learning regarding NeRF.

[0050] A specific example of the process shown in FIG. 8 is shown in FIG. 9. FIG. 9 is a diagram showing a specific example of the error calculation according to the present disclosure.

[0051] In the example shown in FIG. 9, the EVS 10 outputs, as the actual device information 81, information that the calculation method is "Linear (linear function)", the positive threshold is "0.5", and the negative threshold is "0.6".

[0052] The image processing device 100, similar to FIG. 6, is a pair of two time points when an event occurs (time point t ｋ－１ and time point t ｋThe difference in the luminance values of the rendering results in ) is obtained. In the example of FIG. 9, since the image processing apparatus 100 has acquired the information that the calculation method is "Linear", when obtaining the difference value, a linear difference value as shown in Equation 82 is taken. That is, the image processing apparatus 100 acquires information regarding event data from the EVS 10, and based on the acquired information, changes the calculation method in NeRF. In this case, the event error is obtained by Equation 83, and the image processing apparatus 100 can optimize the neural network of NeRF by learning to minimize such an error.

[0053] As described above, as shown using FIGS. 1 to 9, the EVS 10 according to the present disclosure can optimize the accuracy of NeRF according to each actual machine even when various actual machines with different event characteristics are used by outputting actual machine information such as a calculation method related to an event. Although details will be described later, the actual machine information such as the calculation method may be output from the EVS 10 to the image processing apparatus 100, or may be input to the image processing apparatus 100 by a user who has obtained the actual machine information.

[0054] (1-3. Configuration of the Image Processing Apparatus According to the Embodiment) Next, the configuration of the image processing apparatus 100 according to the embodiment will be described using FIG. 10. FIG. 10 is a diagram showing a configuration example of the image processing apparatus 100 according to the embodiment.

[0055] As shown in FIG. 10, the image processing apparatus 100 includes a communication unit 110, a storage unit 120, a control unit 130, an input unit 140, and a display unit 150. Note that the image processing apparatus 100 may include an input unit (for example, a touch panel) that receives various operations from a user who operates the image processing apparatus 100 or the like, and a display unit (for example, a liquid crystal display) for displaying various information.

[0056] The communication unit 110 is implemented by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to the network N (Internet, NFC (Near field communication), Bluetooth, etc.) either wired or wirelessly, and transmits and receives information to and from the EVS10 or the like via the network N. When the EVS10 is mounted inside the image processing apparatus 100, the EVS10 and the image processing apparatus 100 may be connected by the data bus 14 as shown in FIG. 1.

[0057] The storage unit 120 is implemented by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. As shown in FIG. 10, the storage unit 120 includes a real machine information storage unit 121 and a model storage unit 122.

[0058] The real machine information storage unit 121 stores information regarding the event characteristics of the EVS10 and the like. Specifically, the real machine information storage unit 121 stores data such as the calculation method, the positive threshold value, and the negative threshold value when the EVS10 detects an event.

[0059] The model storage unit 122 stores a learned model (NN) that is learned when NeRF is executed. For example, the model is generated for each object to which NeRF is applied and is optimized through learning processes such as minimizing the loss of RGB images and event data.

[0060] The control unit 130 is realized, for example, when a program (for example, an image processing program according to the present disclosure) stored inside the image processing apparatus 100 is executed using a work area such as a RAM (Random Access Memory) by a CPU (Central Processing Unit), a MPU (Micro Processing Unit), or the like. Further, the control unit 130 is a controller and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The specific configuration of the control unit 130 will be described in detail below in FIG. 11 and subsequent figures.

[0061] The input unit 140 receives the input of various information. For example, the input unit 140 receives the input of actual machine information of the EVS 10 and the like via an input device such as a UI (User Interface) operable by the user or a keyboard.

[0062] The display unit 150 displays the information output by the image processing apparatus 100. For example, the display unit 150 is a liquid crystal display or the like built in the image processing apparatus 100 or connected to the image processing apparatus 100.

[0063] (1-4. First Configuration Example of Control Unit) Next, a specific configuration example of the control unit 130 will be shown using FIG. 11. FIG. 11 is a diagram showing a first configuration example of the control unit 130. In the first configuration example, the configuration of the control unit 130 when the user inputs the actual machine information of the EVS 10 is shown.

[0064] As shown in FIG. 11, the control unit 130 includes a data acquisition unit 131, a model construction unit 132, and an actual machine information setting unit 137, and realizes or executes the functions and operations of information processing described below. Further, the model construction unit 132 includes a camera position estimation unit 133, a ray calculation unit 134, an event ray calculation unit 1341, a rendering unit 135, and a learning unit 136. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 11, and any other configuration may be used as long as it performs the information processing described later.

[0065] The data acquisition unit 131 acquires various information. For example, the data acquisition unit 131 acquires the RGB image and event data output from the EVS 10, etc.

[0066] The camera position estimation unit 133 estimates the camera position when each data is obtained based on the RGB image and event data.

[0067] The ray calculation unit 134 calculates a ray passing through a predetermined pixel and the camera position estimated by the camera position estimation unit 133. For example, the event ray calculation unit 1341 calculates a ray passing through the same pixel in two pieces of data that are a pair where an event has occurred.

[0068] The rendering unit 135 obtains the color data of each point using a neural network that takes the coordinates corresponding to the point and the camera position as inputs and outputs the color data at the coordinates for each point sampled on the ray calculated by the event ray calculation unit 1341. Then, the rendering unit 135 collects and renders the color information of a plurality of points to obtain the final pixel value.

[0069] The learning unit 136 learns a machine learning model (neural network) so as to minimize the error (loss), which is the difference between the rendering result in the real space and the rendering result in the NeRF space.

[0070] During the learning process of the learning unit 136, the actual machine information setting unit 137 provides the learning unit 136 with actual machine information such as the calculation method of the event threshold. Note that the actual machine information setting unit 137 may be a processing unit that acquires the actual machine information output from the EVS 10 and provides the acquired information to the learning unit 136, or may be an input unit that receives an input of actual machine information from the user.

[0071] Next, the processing of each unit constituting the above-described image processing apparatus 100 will be described in detail along the flow using FIG. 12. FIG. 12 is a flowchart showing the flow of processing in the first configuration example of the control unit.

[0072] The image processing apparatus 100 acquires data such as RGB images and event data (step S11). When the image processing apparatus 100 acquires actual machine information from the EVS 10 or a user or the like at this time, it sets the actual machine information (step S17).

[0073] For example, as actual machine information regarding the EVS 10 used for shooting, the user inputs, via, for example, a GUI (Graphical User Interface) provided by the image processing apparatus 100, a predetermined setting file, or the like, information such as event thresholds (bright transition threshold and dark transition threshold), a method for calculating the event threshold, and information on pixel positions where an event can be output.

[0074] Subsequently, the image processing apparatus 100 estimates the camera position regarding the RGB image and the event data (step S12). This process will be described using FIG. 13. FIG. 13 is a flowchart regarding the camera position estimation process.

[0075] First, the image processing apparatus 100 estimates the camera position when the RGB image was taken from the feature amounts of the RGB image using a known method (step S121). Subsequently, the image processing apparatus 100 estimates the camera position for each time stamp when an event was recorded (step S122). At this time, regarding the camera position during the shooting of the RGB image, the image processing apparatus 100 may perform spline interpolation or the like based on the shooting time of the RGB image and the shooting time of the event so that the camera positions are smoothly connected.

[0076] Returning to FIG. 12, after estimating the camera position, the image processing apparatus 100 calculates the event ray (step S13). This process will be described using FIG. 14. FIG. 14 is a flowchart regarding the event ray calculation process.

[0077] First, the image processing apparatus 100 selects two pieces of data that are a pair in which an event has occurred (step S131). Specifically, the image processing apparatus 100 extracts a pair of events that have occurred continuously at the same pixel (referred to as the "first event" and the "second event") and selects the extracted pair.

[0078] Then, the image processing apparatus 100 acquires the camera positions at each time point corresponding to the selected event (step S132). Subsequently, the image processing apparatus 100 calculates a light ray that passes through the coordinate value of the event and the camera position (step S133). Note that the image processing apparatus 100 sequentially calculates light rays for the first event and the second event, and thus loops the processes of step S132 and step S133 the necessary number of times.

[0079] Returning to FIG. 12, the image processing apparatus 100 samples the coordinates on the calculated light ray (step S14). The image processing apparatus 100 acquires the color of each sampled coordinate value using a neural network (step S15). The image processing apparatus 100 loops the process of step S15 the necessary number of times to acquire colors for the number of coordinates.

[0080] Subsequently, the image processing apparatus 100 performs volume rendering based on the obtained color information (step S16). Since the processes up to step S16 require processing for two light rays (processing related to the first event and the second event), the image processing apparatus 100 loops the processes from step S13 to step S16 the necessary number of times.

[0081] Subsequently, the image processing apparatus 100 calculates an error based on the rendering result (step S18). This process will be described using FIG. 15. FIG. 15 is a flowchart related to error calculation.

[0082] First, the image processing apparatus 100 reads out the actual machine information acquired in step S17 (step S181). Then, the image processing apparatus 100 calculates the difference value of the rendering result based on the actual machine information (step S182). Subsequently, the image processing apparatus 100 calculates an error based on the difference value and the threshold value of the actual machine information (step S183).

[0083] The above process will be described using FIG. 16. FIG. 16 is a diagram for explaining the error calculation according to the embodiment.

[0084] Specifically, the difference value of rendering results is the difference value between the first rendering result based on the first event and the second rendering result based on the second event. In the example of FIG. 16, pixel u included in image 73 ｋ and the pixel value of pixel u ｋ included in image 74. The image processing apparatus 100 calculates the difference value of these rendering results. That is, the mathematical formula shown in FIG. 16 (the following formula (1)) corresponds to step S182 shown in FIG. 15.

[0085]

[0086] Then, the image processing apparatus 100 obtains the difference L k between the obtained error and the threshold value (p evs C) used for event detection. That is, the mathematical formula shown in FIG. 16 (the following formula (2)) corresponds to step S183 shown in FIG. 15.

[0087]

[0088] Here, regarding the difference value calculation of rendering results (step S182) and error calculation (step S183), in addition to the example shown in FIG. 16, there may be modified examples according to the calculation method of actual machine information and threshold value setting.

[0089] For example, regarding the difference value calculation, when the calculation method of the event threshold value is the natural logarithm, the image processing apparatus 100 may apply the following formula (3).

[0090]

[0091] When the threshold value calculation method is the common logarithm, the image processing apparatus 100 uses the common logarithm instead of the natural logarithm in the following formula (3). Also, regarding the difference value calculation, when the calculation method of the event threshold value is the linear threshold value, the image processing apparatus 100 may apply the following formula (4).

[0092]

[0093] As described above, as the threshold calculation method, at least one of a linear threshold, a natural logarithm threshold, or a common logarithm threshold can be adopted. Also, regarding error calculation, when only one value is set as the event threshold, the image processing apparatus 100 can apply the following formula (5).

[0094]

[0095] Also, regarding error calculation, when two values, that is, a bright transition threshold and a dark transition threshold, are set as the event threshold, the image processing apparatus 100 can apply the following formula (6).

[0096]

[0097] Returning to FIG. 12, the image processing apparatus 100 updates the weights of the neural network so that the calculated error is minimized (step S19). Thus, the image processing apparatus 100 can perform optimal difference value calculation and error calculation according to the actual machine information, and update the model based on the result, so that high-precision learning suitable for the actual machine information can be performed.

[0098] (1-5. Second configuration example of the control unit) Next, an example in which the image processing apparatus 100 acquires information regarding event filtering and the luminance difference value between events as actual machine information is shown. FIG. 17 is a diagram showing an outline of image processing in the second configuration example of the control unit.

[0099] In such an example, it is assumed that an event filter is applied to the EVS 10. An event filter means that when the same event continuously occurs in the same pixel in order to suppress the frequent detection of events, only some events are filtered and output. By filtering, the data amount of event data output from the EVS 10 can be reduced. Therefore, in the EVS 10, the event filter is frequently used.

[0100] However, when the event filter is effective, there is a possibility that the luminance difference between two events does not match the event threshold. Therefore, it may affect the learning process using the event threshold.

[0101] In this regard, in the image processing of the present disclosure, for each event, the number of filtered events or the luminance difference from the previously output event is output from EVS10, and such output is read on the NeRF side and reflected in the error calculation process in NeRF. Thereby, even when the event filter is effective, according to the image processing of the present disclosure, highly accurate learning can be performed.

[0102] FIG. 20 shows time-series data 200 illustrating an example of event detection when an event filter is applied. As shown in FIG. 20, the time-series data 200 includes information regarding the number of events indicating information such as how many events such as positive events are filtered. Further, the time-series data 200 includes information on the difference (logarithmic difference in the example of FIG. 20) between the luminance value when the previous event was detected and the luminance value in the event detected next excluding the filtered events. In the following description, the event that is the starting point when calculating the luminance difference value (synonymous with the above-mentioned "previous event") is referred to as the "first event", and the event to be measured for the difference from the starting event (the above-mentioned "event detected next") is referred to as the "second event".

[0103] As shown in FIG. 20, when the luminance value is indicated by "C p " (positive) or "C n " (negative), the event error in NeRF is expressed by Equation 210. This is based on the premise that, similar to what was shown in the first configuration example, the difference value of the rendering result coincides with the event error (here, the luminance difference value between events). That is, in the second configuration example, the image processing apparatus 100 performs learning regarding NeRF based on the number of filtered events, the luminance difference value, etc., rather than the calculation method of the event threshold value.

[0104] Regarding Equation 210, when generalizing the event error in NeRF when an event filter is applied, it can be expressed as the following formula (7). In the following formula (7), the threshold value corresponding to the polarity p k-1 is "C k-1 ", and the threshold value corresponding to the polarity p k is "C kshall be "」.

[0105]

[0106] In this way, even when the event filter is applied, the image processing apparatus 100 can perform the learning process of NeRF using the event data. As a result, the image processing apparatus 100 can realize a more accurate NeRF while achieving further low power by reducing redundant event outputs.

[0107] While showing the second configuration example of the control unit 130, the above process will be described. FIG. 18 is a diagram showing the second configuration example of the control unit 130 of the embodiment. Note that the description of the same configuration and process as in FIG. 11 will be omitted.

[0108] In the second configuration example, since it is necessary to obtain the number of filterings for each event, the image processing apparatus 100 basically obtains information by analyzing the data sent from the EVS 10 instead of the input from the user. For this reason, as shown in FIG. 18, in the second configuration example, the control unit 130 has a real machine information analysis unit 138.

[0109] The real machine information analysis unit 138 analyzes the real machine information from the event data acquired from the EVS 10. Specifically, the real machine information analysis unit 138 analyzes the data added by the EVS 10 as additional information for each event, and obtains information such as the number of events indicating the number of filtered events and the luminance difference value between the first event and the second event. In the second configuration example, since the number of filterings for each event is required, the data sent from the sensor is analyzed to obtain information.

[0110] The learning unit 136 performs learning processes such as error calculation and weight update using the number of events and the luminance difference value analyzed by the real machine information analysis unit 138.

[0111] (1-6. Third Configuration Example of Control Unit) Next, an example in which the image processing apparatus 100 performs learning using information on pixels that detect events will be described. FIG. 19 is a diagram showing an overview of image processing in the third configuration example of the control unit.

[0112] In the third configuration example, the image processing apparatus 100 is characterized in that, in learning regarding event data, it uses not only information of pixels capable of detecting an event but also information of pixels where no event is detected for learning.

[0113] As described above, the image processing apparatus 100 learns with the difference between the luminance difference value of the rendering result in the NeRF space and an event threshold value or the like as an error. This is based on the premise that "an event occurs in the NeRF space as well" in "pixels and timings where an event occurs in a series of processes of photographing an object". In other words, since the image processing apparatus 100 does not evaluate pixels where "no event occurs" in the photographing scene at all, there is a possibility that a space that causes an event to occur excessively may be learned only by this error term (specifically, the right term of the above formula (2)).

[0114] Therefore, in the third configuration example, the image processing apparatus 100 adds an error term for learning so that "no event occurs in the NeRF space" in an area where no event occurs in the photographing scene, and performs learning. For example, the image processing apparatus 100 performs learning using the following formula (8).

[0115]

[0116] The above formula (8) is a mathematical formula for obtaining an error of "no event occurs" when the difference is shown as a logarithmic luminance difference. For example, the image processing apparatus 100 uses the above formula (8) to perform an operation such that if the logarithmic luminance difference of the same pixel at a timing when no event occurs is equal to or less than a predetermined threshold value "C", it is 0, and if it exceeds the threshold value "C", an error occurs.

[0117] One of the reasons why learning is also performed for pixels where no event occurs is that the resolution of the event is different from the resolution of RGB pixels. For example, in a hybrid EVS such as the EVS10 capable of acquiring event data and RGB, the resolution of the event is, for example, about one-fourth of the RGB image resolution after development. Hereinafter, for the sake of distinction, pixels capable of event detection are referred to as "event pixels".

[0118] Regarding the above, it will be described using FIG. 19 and the following figures. FIG. 19 conceptually shows the difference between the configuration of the image sensor for obtaining the RGB image and the configuration of the image sensor for obtaining event data in the EVS10. As described above, the resolution of the sensor for detecting RGB pixels is higher than the resolution of the sensor for detecting events. Therefore, as shown in FIG. 19, the RGB pixel group 220 has a larger number of pixels than the event pixel group 221.

[0119] For this reason, event pixels are arranged, for example, at a ratio of one to four RGB pixels as shown in arrangement 222. Actually, as shown in arrangement 223, an event pixel is assigned to any one of the four RGB pixels (two columns and two rows).

[0120] The occurrence of events in the arrangement shown in FIG. 19 will be described. FIG. 20 is a diagram for explaining the occurrence of events.

[0121] In the arrangement 230 shown in FIG. 20, the pixel 231 indicated by blank is not an event pixel, so it is unknown whether an event has occurred. On the other hand, it has been detected that a positive event has occurred in pixel 232. Also, pixel 233 is a pixel where an event can be detected, but no event has occurred.

[0122] This indicates that in an area other than event pixels like pixel 231, there is no correct data regarding the occurrence of events because events have not been measured in the first place. And if the image processing device 100 regards the area without correct data of events as "no event has occurred" and calculates the non-occurrence error of events, it may have an adverse effect on the learning accuracy. Specifically, as shown in the arrangement 240 of FIG. 20, when the pixels sampled for learning are pixels where the correct data regarding the occurrence of events is unknown, it may have an adverse effect on the accuracy of image reconstruction in the NeRF space.

[0123] Therefore, the EVS10 and the image processing apparatus 100 according to the present disclosure solve the above problem by using pixel information indicating which pixel an event detection is performed on. This point will be described with reference to FIG. 21. FIG. 21 is a diagram for explaining the output of pixel position information.

[0124] In FIG. 21, an arrangement 230 is shown as in FIG. 20. The arrangement 230 indicates information on whether a positive event, a negative event, or no event has occurred for each pixel.

[0125] In the example of FIG. 21, it is assumed that the EVS10 outputs information indicating whether a pixel is an event-detectable pixel. In this case, although no event has occurred in pixel 233, since pixel 233 has information indicating that it is an "event-detectable pixel", it can be understood that pixel 233 is not a pixel where an event has not been detected but is a pixel where it is certain that no event has occurred.

[0126] Then, as shown in arrangement 250, the image processing apparatus 100 can select pixels where it is certain that no event has occurred as pixels to be sampled, excluding pixels where an event is detectable or pixels whose status is unknown. That is, the image processing apparatus 100 can target only non-event-occurring pixels among event pixels for sampling of non-event-occurring errors with certainty. The pixel position information is generated as additional information including information indicating a coordinate range in which an event can be detected from an event detection unit 22, based on the coordinate position of a luminance image obtained by a luminance detection unit 21 that detects the luminance value of light received by a photodiode. The coordinate range may be any information as long as it is information for specifying event pixels, and examples include a list of coordinates, a list of rows and columns, or a group of coordinate values indicating the endpoints of a range.

[0127] According to the third configuration example, even when the resolutions of the RGB image and the event are different, the image processing apparatus 100 can realize high-precision image reconstruction without reducing the resolution. In addition, even when ROI (Region of Interest) or decimation processing is set and events are not output from all pixels, the image processing apparatus 100 can perform correct calculations by similarly providing information on the output area.

[0128] While showing the third configuration example of the control unit 130, the above process will be described. FIG. 22 is a diagram showing the third configuration example of the control unit 130 of the embodiment. Note that the description of the same configuration and process as in FIGS. 11 and 18 will be omitted.

[0129] In the third configuration example, the control unit 130 includes a real machine information analysis unit 139. In the third configuration example, the real machine information analysis unit 139 receives information on the output area such as the position of event pixels as additional information from the EVS 10 or user input. Then, the real machine information analysis unit 139 provides the received information to the ray calculation unit 134 and the learning unit 136.

[0130] In the third configuration example, the ray calculation unit 134 includes a non-event ray calculation unit 1342. The non-event ray calculation unit 1342 can calculate a ray for learning pixels where an event has surely not occurred by calculating a ray based on information indicating the position of event pixels.

[0131] Next, the processing flow of the third configuration example will be described using FIGS. 23 and below. FIG. 23 is a flowchart showing the processing flow in the third configuration example of the control unit. Note that the description of the same process as in FIG. 12 will be omitted.

[0132] When the image processing apparatus 100 acquires information on event pixels as real machine information, it calculates a non-event ray based on such information (step S21). The image processing apparatus 100 samples the coordinates on the ray (step S22), acquires the color of each coordinate (step S23), and performs volume rendering (step S24). Thereafter, the image processing apparatus 100 performs error calculation regarding the difference value (step S25).

[0133] Then, based on the error calculation related to the event detection in step S18 and the error calculation related to the non-event detection in step S25, the image processing apparatus 100 performs weight update (step S26). Thus, by learning using the information of events and non-events, the image processing apparatus 100 can improve the accuracy of the model (the accuracy of image reconstruction).

[0134] FIG. 24 is a flowchart related to the calculation of non-event light rays. The image processing apparatus 100 reads actual machine information (step S211), specifies event pixels based on such information, and selects non-event pairs (step S212). Specifically, based on the information of pixel positions where events can be output, the image processing apparatus 100 selects pixels where no event has occurred for a certain period, and selects a pair of a first shooting time and a second shooting time, which are arbitrary shooting times within that certain period. The image processing apparatus 100 may select pairs randomly.

[0135] Then, similar to steps S132 and S133 shown in FIG. 14, the image processing apparatus 100 acquires the camera position corresponding to the non-event shooting time (step S213), and calculates the light rays passing through the non-event coordinate values and the camera position (step S214). The image processing apparatus 100 executes steps S213 and S214 for each of the first event and the second event.

[0136] [[ID=1第十一条]] FIG. 25 is a flowchart related to the error calculation of non-events. The image processing apparatus 100 reads actual machine information such as the calculation methods of the bright transition threshold, the dark transition threshold, and the event threshold (step S251). Subsequently, similar to step S182 shown in FIG. 15, the image processing apparatus 100 calculates a difference value (step S252).

[0137] Then, the image processing apparatus 100 performs an error calculation related to non-events using the above formula (8) (step S253).

[0138] (1-7. Data Structure) Next, the data structure of the data output by the EVS 10 according to the embodiment will be described. FIG. 26 is a diagram (1) showing an example of the data structure of the data output by the EVS 10.

[0139] The output data 300 shown in FIG. 26 conceptually represents the data output from the EVS 10. The output data 300 has, for example, a data structure compliant with the MIPI standard. The output data 300 has embedded data 310 including various types of information such as actual machine information.

[0140] As described above, the EVS 10 of the present disclosure embeds information indicating the characteristics of the entire sensor, which is added to event data and the like, into the embedded data 310 and outputs it to an external device or the like. Specifically, the EVS 10 outputs a threshold calculation method 311, a positive threshold 312 for determining a positive event, and a negative threshold 313 for determining a negative event as additional information. Further, the EVS 10 may output pixel position information 314 such as which position the event pixel corresponds to, as shown in FIGS. 20 and 21, as additional information.

[0141] Note that the EVS 10 may generate a frame that combines a plurality of events that occur at the same time or within a certain period. In this case, the EVS 10 may output a calculation method for each frame.

[0142] Further, the EVS 10 may output information regarding an event filter. This point will be described using FIG. 27. FIG. 27 is a diagram (2) showing an example of the data structure of the data output by the EVS 10.

[0143] As shown in FIG. 27, the output data 300 may have a region 320 for recording the polarity (positive or negative) of an event and a region 321 for recording the number of event filterings in the event data. The image processing apparatus 100 can determine the number of events that have been filtered by acquiring such information.

[0144] Note that the data structures shown in FIGS. 26 and 27 are just examples, and the additional information added to the event data (in other words, the actual machine information of EVS10) can take various types and forms.

[0145] (1-8. Variations according to the embodiment) The information processing according to the embodiment described above may involve various variations. The variations of the embodiment will be described below.

[0146] The event data shown in the present disclosure may be obtained in any format. For example, the event data may be output according to the difference with respect to luminance (photon rate) using a SPAD (Single Photon Avalanche Diode).

[0147] Also, in the above-described embodiment, an example of performing learning using an error term related to event data in NeRF was shown. However, the image processing apparatus 100 may add a normal NeRF error function using RGB data in addition to the event data to learn the model. Thereby, the image processing apparatus 100 can obtain a NeRF that can accurately restore color information. Regarding the RGB data and the event data, the image processing apparatus 100 may use those obtained from one stacked EVS10, or may use those obtained from different sensors after alignment.

[0148] Also, when the user inputs actual machine information to the image processing apparatus 100, the user may not only simply input the setting information of the EVS10, but also input information arbitrarily adjusted by the user. In this case, the image processing apparatus 100 can change the calculation method in the NeRF according to the information input from the user.

[0149] Further, as a configuration of the image processing apparatus 100, instead of the actual machine information setting unit 137, an error calculation unit may be implemented as a plugin by the user. That is, the user implements, as an error calculation unit, a process of receiving two rendering results, calculating an error using them, and returning the result as a plugin. According to such a configuration, the user can realize a process corresponding to sensors of any threshold calculation method regardless of the sensor settings.

[0150] Further, the output data of the EVS 10 of the present disclosure may be used not only for NeRF but also for any program or application. For example, the output data of the EVS 10 can be used in software or applications that image it in a form such as a count image having the number of occurrences of events within a predetermined period as the number of pixels based on the event data. Specifically, such an application can generate a count image in which not only the number of occurrences of events but also the number of event filters is added as a pixel value by using the output data of the EVS 10.

[0151] (2. Other Embodiments) The processes according to the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0152] Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0153] Further, each component of each of the illustrated apparatuses is conceptually functional and does not necessarily need to be physically configured as illustrated. That is, the specific form of distribution and integration of each apparatus is not limited to that illustrated, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage situations.

[0154] Further, the above-described embodiments and modifications can be appropriately combined within a range that does not conflict with the processing content.

[0155] Further, the effects described in this specification are merely examples and are not limited, and there may be other effects.

[0156] (3. Effects of the image sensor and image processing apparatus according to the present disclosure) As described above, the image sensor (EVS10 in the embodiment) according to the present disclosure includes an event detection unit (event detection unit 22 in the embodiment), an additional information generation unit (additional information generation unit 23 in the embodiment), and a data output unit (data output unit 24 in the embodiment). The event detection unit detects the occurrence of an event that is a change in the luminance of light received by the photodiode. The additional information generation unit generates additional information including a calculation method of an event threshold used when detecting the detected event, together with event data indicating the content of the detected event. The data output unit outputs the additional information.

[0157] Thus, the image sensor according to the present disclosure is characterized in that not only event data but also a calculation method of an event threshold used for event detection is output to the outside. For this reason, the image sensor can perform highly accurate learning based on the calculation method in a learning process using event data such as NeRF. Thereby, the image sensor can improve the learning accuracy of the neural network.

[0158] Further, the additional information generation unit generates additional information including at least one of a linear threshold, a natural logarithm threshold, or a common logarithm threshold as the calculation method.

[0159] Thus, the image sensor can enable appropriate error calculation in subsequent processing such as NeRF by outputting a specific method adopted for calculating the event threshold.

[0160] Further, the event detection unit generates a frame that combines a plurality of events that have occurred at the same time or within a certain period. The data output unit outputs the calculation method for each frame.

[0161] Thus, the image sensor can improve transmission performance or reduce processing load by outputting data obtained by summarizing events.

[0162] Further, the event detection unit may perform event filtering to detect only a part of events that occur continuously in the same pixel.

[0163] For example, in event filtering, the event detection unit detects the number of events that were detected but not output between the previous event output from the same pixel and the occurrence of the next event. The additional information generation unit generates additional information including the number of events filtered at the time of event detection.

[0164] Thus, since the image sensor outputs the number of filtered events, error calculation and the like can be performed without causing the problem that the luminance difference between two events does not match the event threshold.

[0165] Further, in event filtering, the event detection unit detects the luminance difference between the luminance value when the previous event was output from the same pixel and the luminance value when the next event is output. The additional information generation unit generates additional information including the luminance difference.

[0166] Thus, the image sensor may directly output the luminance difference value between pixels used for determining event errors instead of the number of filtered events. Thereby, the image sensor can perform error calculation and the like without causing the problem that the luminance difference between two events does not match the event threshold.

[0167] Further, the image sensor further includes a luminance detection unit that detects the luminance value of the light received by the photodiode. The additional information generation unit generates, as additional information, information indicating a coordinate range in which an event can be detected from the event detection unit, based on the coordinate position of the luminance image obtained from the luminance detection unit.

[0168] The additional information generation unit generates, as the coordinate range, at least any one of information of a list of coordinates, a list of rows and columns, or a group of coordinate values indicating the endpoints of the range.

[0169] Thus, by adding the position information of the event pixel as actual machine information, the image sensor can distinguish whether a certain pixel simply did not detect an event or was a pixel that could not detect an event in the first place. By using such information, the subsequent device can perform, for example, more accurate learning.

[0170] Further, the image processing apparatus (image processing apparatus 100 in the embodiment) according to the present disclosure includes an acquisition unit (data acquisition unit 131 in the embodiment) and a model construction unit (model construction unit 132 in the embodiment). The acquisition unit acquires event data indicating the content of the event detected by the image sensor and actual machine information that is information regarding the settings when the event data is obtained. The model construction unit constructs a model capable of generating an image corresponding to a given arbitrary camera position based on the actual machine information. Note that the actual machine information includes the calculation method of the event threshold used when the image sensor detects an event.

[0171] Thus, the image processing apparatus according to the present disclosure acquires not only the event data but also the calculation method of the event threshold used for event detection, and performs learning of the model based on the acquired information. Thereby, the image processing apparatus can perform highly accurate learning based on the calculation method in the learning process using event data such as NeRF. Thereby, the image processing apparatus can improve the learning accuracy of the neural network.

[0172] Further, the actual machine information may further include information on the light transition threshold and the dark transition threshold used when the image sensor detects an event.

[0173] Further, the model construction unit calculates a ray corresponding to the event based on the event data, samples points on the ray, and renders based on a machine learning model that estimates the pixel information of the sampled points, and learns the machine learning model based on the rendering result.

[0174] Specifically, the model construction unit calculates a first ray that passes through a pixel where a first event occurred and a second ray that passes through the pixel where a second event occurred at a time after the first event. Furthermore, the model construction unit calculates a difference between rendering results of a first point corresponding to a pixel on the first ray and a second point corresponding to a pixel on the second ray, and trains a machine learning model based on the difference between the rendering results.

[0175] For example, the model construction unit calculates the brightness difference value obtained from the rendering results of the first point and the second point and the error obtained from the brightness difference value based on either the event threshold calculation method included in the actual machine information or the event's brightening threshold and darkening threshold.

[0176] In this way, the image processing device can perform highly accurate learning by using information such as the event threshold calculation method and the event light change threshold and dark change threshold in the process of constructing a model to estimate pixel information such as NeRF.

[0177] The actual machine information may further include information on pixel positions where an event may be detected, which was used when the image sensor detected the event. In this case, the model construction unit selects pixels where no event has occurred for a certain period of time based on the pixel position information, selects a pair of a first image capture time and a second image capture time that are arbitrary image capture times within the certain period, and calculates light rays passing through each pixel of the selected pair.

[0178] In this way, the image processing device can learn a model using not only information on whether an event has occurred but also information on pixels where an event has definitely not occurred, thereby enabling the image processing device to build a model with higher accuracy.

[0179] The actual machine information may further include information that event filtering has been performed to detect only a portion of events that have consecutively occurred at the same pixel, and the number of events that were detected but not output during the event filtering period from the previously output event to the next event occurrence at the same pixel. In this case, the model construction unit calculates the brightness difference values ​​obtained from the rendering results for the first point and the second point and the errors obtained from the brightness difference values ​​based on the event threshold calculation method, the event brightening threshold and darkening threshold included in the actual machine information, or the number of filtered events.

[0180] In this way, the image processing device can also learn a model based on the number of filtered events, which allows the image processing device to perform highly accurate learning even when using an image sensor that operates with low power and suppresses event output.

[0181] The actual machine information may further include information that event filtering has been performed to detect only a portion of events that have consecutively occurred at the same pixel, and that the event filtering has included a luminance difference between the luminance value when the previous event was output at the same pixel and the luminance value when the next event is output. In this case, the model construction unit calculates the luminance difference values ​​obtained from the rendering results of the first point and the second point and the error obtained from the luminance difference values ​​based on the event threshold calculation method included in the actual machine information, the event brightening threshold and darkening threshold, or the luminance difference between the luminance value when the previous event was output at the same pixel and the luminance value when the next event is output.

[0182] In this way, the image processing device can also train a model using the brightness difference values ​​between filtered events. This method also allows the image processing device to perform highly accurate training, similar to training using the number of filtered events.

[0183] The acquisition unit may also acquire the actual machine information through an input operation by a user. That is, the image processing device may be configured to directly receive output from the image sensor, or may be configured to accept input from a user, etc. By accepting input from the user, for example, the image processing device can acquire actual machine information adjusted by the user in accordance with the characteristics of the image sensor, etc., thereby performing appropriate learning that is more tailored to the characteristics of the image sensor.

[0184] (4. Hardware Configuration) Information devices such as the image processing device 100 and EVS 10 according to the above-described embodiments are realized by a computer 1000 configured as shown in FIG. 28 , for example. The image processing device 100 according to the embodiments will be described below as an example. FIG. 28 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the image processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0185] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0186] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0187] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a conversion program according to the present disclosure, which is an example of program data 1450.

[0188] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0189] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0190] For example, when the computer 1000 functions as the image processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an image processing program loaded onto the RAM 1200, thereby realizing the functions of the control unit 130, etc. The image processing program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0191] The present technology may also be configured as follows. (1) An image sensor including: an event detection unit that detects the occurrence of an event, which is a change in luminance of light received by a photodiode; an additional information generation unit that generates additional information including a calculation method of an event threshold used to detect the event, along with event data indicating the content of the detected event; and a data output unit that outputs the additional information. (2) The image sensor according to (1), wherein the additional information generation unit generates additional information including at least one of a linear threshold, a natural logarithm threshold, or a common logarithm threshold as the calculation method. (3) The image sensor according to (2), wherein the event detection unit generates frames that group together multiple events that occurred at the same time or within a certain period, and the data output unit outputs the calculation method for each frame. (4) The image sensor according to any one of (1) to (3), wherein the event detection unit performs event filtering to detect only a portion of events that occurred consecutively at the same pixel. (5) The image sensor according to (4), wherein the event detection unit detects the number of events that were detected but not output between the previously output event and the occurrence of the next event at the same pixel in the event filtering, and the additional information generation unit generates additional information including the number of events filtered at the time of event detection. (6) The image sensor according to (4) or (5), wherein the event detection unit detects a luminance difference between a luminance value at the time the previous event was output at the same pixel and a luminance value at the time the next event is output at the same pixel in the event filtering, and the additional information generation unit generates additional information including the luminance difference. (7) The image sensor according to any one of (1) to (6), further comprising a luminance detection unit that detects a luminance value of light received by the photodiode, and the additional information generation unit generates the additional information including information indicating a coordinate range in which an event can be detected by the event detection unit, based on a coordinate position of a luminance image obtained from the luminance detection unit.(8) The image sensor according to (7), wherein the additional information generation unit generates, as the coordinate range, at least one of information on a list of coordinates, a list of rows and columns, or a group of coordinate values ​​indicating end points of a range. (9) An image processing device comprising: an acquisition unit that acquires event data indicating details of an event detected by the image sensor and actual machine information that is information on settings when the event data was obtained; and a model construction unit that constructs, based on the actual machine information, a model that can generate an image corresponding to an arbitrary camera position when the camera position is given, wherein the actual machine information includes a calculation method for an event threshold used when the image sensor detects the event. (10) The image processing device according to (9), wherein the actual machine information further includes information on a light threshold and a dark threshold of the event used when the image sensor detects the event. (11) The image processing device according to (10), wherein the model construction unit calculates a ray corresponding to the event based on the event data, samples points on the ray, and performs rendering based on a machine learning model that estimates pixel information of the sampled points, and trains the machine learning model based on a result of the rendering. (12) The image processing device according to (11), wherein the model construction unit calculates a first ray that passes through a pixel where a first event occurred and a second ray that passes through the pixel where a second event occurred at a time after the first event, calculates a difference between rendering results of a first point corresponding to the pixel on the first ray and a second point corresponding to the pixel on the second ray, and trains the machine learning model based on the difference between the rendering results. (13) The image processing device according to (12), wherein the model construction unit calculates a brightness difference value obtained from the rendering results of the first point and the second point and an error obtained from the brightness difference value based on either a calculation method of the event threshold included in the actual machine information or a brightening threshold and a darkening threshold of the event.(14) The image processing device according to any one of (10) to (13), wherein the actual machine information further includes information on pixel positions where an event can be detected, which was used when the image sensor detected the event, and the model construction unit selects pixels where the event has not occurred for a certain period based on the pixel position information, and selects a pair of a first shooting time and a second shooting time, which are arbitrary shooting times within the certain period, and calculates light rays passing through each pixel of the selected pair. (15) The image processing device according to any one of (12) to (14), wherein the actual machine information further includes that event filtering has been performed to detect only a portion of events that have occurred consecutively at the same pixel, and the number of events that were detected but not output in the event filtering between the previously output event and the occurrence of the next event at the same pixel, and the model construction unit calculates a brightness difference value obtained from the rendering results of each of the first point and the second point and an error obtained from the brightness difference value based on the calculation method of the event threshold, the brightening threshold and darkening threshold of the event, or the number of filtered events, which are included in the actual machine information. (16) The image processing device according to any one of (12) to (15), wherein the actual machine information includes that event filtering has been executed to detect only a portion of events that have occurred consecutively at the same pixel, and that the event filtering further includes a luminance difference between a luminance value when a previous event was output at the same pixel and a luminance value when a next event is output, and the model construction unit calculates a luminance difference value obtained from each rendering result of the first point and the second point and an error obtained from the luminance difference value based on any one of a calculation method of the event threshold, a lightening threshold and a darkening threshold of the event, included in the actual machine information, or a luminance difference between a luminance value when a previous event was output at the same pixel and a luminance value when a next event is output. (17) The image processing device according to any one of (9) to (16), wherein the acquisition unit acquires the actual machine information through an input operation by a user.(18) An image processing method including a computer acquiring event data indicating the content of an event detected by an image sensor and actual machine information which is information about the settings when the event data was obtained, and constructing a model based on the actual machine information that can generate an image corresponding to an arbitrary camera position when the camera position is given, wherein the actual machine information includes a calculation method for an event threshold used when the image sensor detects the event.

[0192] 10 EVS 21 Brightness detection unit 22 Event detection unit 23 Additional information generation unit 24 Data output unit 100 Image processing device 110 Communication unit 120 Storage unit 121 Real machine information storage unit 122 Model storage unit 130 Control unit 131 Data acquisition unit 132 Model construction unit 133 Camera position estimation unit 134 Ray calculation unit 135 Rendering unit 136 Learning unit 137 Real machine information setting unit 138 Real machine information analysis unit

Claims

1. An image sensor comprising: an event detection unit that detects the occurrence of an event, which is a change in the brightness of light received by a photodiode; an additional information generation unit that generates additional information including event data indicating the content of the detected event and a calculation method for the event threshold used to detect the event; and a data output unit that outputs the additional information.

2. The image sensor according to claim 1, wherein the additional information generating unit generates additional information including at least one of a linear threshold, a natural logarithm threshold, and a common logarithm threshold as the calculation method.

3. The image sensor according to claim 2, wherein the event detection unit generates a frame that groups together multiple events that occurred at the same time or within a certain period of time, and the data output unit outputs the calculation method for each frame.

4. The image sensor according to claim 1, wherein the event detection unit performs event filtering to detect only a portion of events that occur consecutively at the same pixel.

5. The image sensor according to claim 4, wherein the event detection unit detects the number of events that were detected but not output between the previous output event and the next event occurrence at the same pixel during the event filtering, and the additional information generation unit generates additional information including the number of filtered events at the time of event detection.

6. The image sensor according to claim 4, wherein the event detection unit detects a luminance difference between a luminance value when a previous event was output from the same pixel and a luminance value when a next event is output from the same pixel in the event filtering, and the additional information generation unit generates additional information including the luminance difference.

7. The image sensor according to claim 1, further comprising a brightness detection unit that detects the brightness value of light received by the photodiode, wherein the additional information generation unit generates the additional information including information indicating a coordinate range in which an event can be detected by the event detection unit, based on the coordinate position of the brightness image obtained from the brightness detection unit.

8. The image sensor according to claim 7, wherein the additional information generating unit generates, as the coordinate range, at least one of information in the form of a list of coordinates, a list of rows and columns, or a group of coordinate values ​​indicating the endpoints of the range.

9. An image processing device comprising: an acquisition unit that acquires event data indicating the content of an event detected by an image sensor and actual machine information, which is information regarding the settings when the event data was obtained; and a model construction unit that constructs a model that can generate an image corresponding to an arbitrary camera position when the camera position is given, based on the actual machine information, wherein the actual machine information includes a calculation method for an event threshold used when the image sensor detects the event.

10. The image processing device according to claim 9, wherein the actual device information further includes information on the light threshold and dark threshold of the event used when the image sensor detected the event.

11. The image processing device described in claim 10, wherein the model construction unit calculates a ray corresponding to the event based on the event data, samples points on the ray, and renders them based on a machine learning model that estimates pixel information of the sampled points, and learns the machine learning model based on the results of the rendering.

12. The image processing device described in claim 11, wherein the model construction unit calculates a first ray passing through a pixel where a first event occurred and a second ray passing through the pixel where a second event occurred at a time after the first event, calculates the difference in rendering results between a first point corresponding to the pixel on the first ray and a second point corresponding to the pixel on the second ray, and learns the machine learning model based on the difference in rendering results.

13. The image processing device described in claim 12, wherein the model construction unit calculates the brightness difference value obtained from the rendering results of each of the first point and the second point and the error obtained from the brightness difference value based on either the calculation method of the event threshold included in the actual machine information or the brightening threshold and darkening threshold of the event.

14. The image processing device described in claim 10, wherein the actual machine information further includes information on pixel positions at which an event can be detected, which was used when the image sensor detected the event, and the model construction unit selects pixels at which the event has not occurred for a certain period of time based on the pixel position information, and selects a pair of a first shooting time and a second shooting time, which are arbitrary shooting times within the certain period of time, and calculates light rays passing through each pixel of the selected pair.

15. The image processing device according to claim 12, wherein the actual machine information further includes information that event filtering has been performed to detect only a portion of events that have occurred consecutively at the same pixel, and the number of events that were detected but not output in the event filtering between the previously output event and the occurrence of the next event at the same pixel, and the model construction unit calculates the brightness difference value obtained from the rendering results of each of the first point and the second point and the error obtained from the brightness difference value based on either the calculation method of the event threshold included in the actual machine information, the brightening threshold and darkening threshold of the event, or the number of filtered events.

16. The image processing device according to claim 12, wherein the actual machine information indicates that event filtering has been performed to detect only a portion of events that have occurred consecutively at the same pixel, and further includes, in the event filtering, a luminance difference between the luminance value when the previous event was output at the same pixel and the luminance value when the next event is output, and the model construction unit calculates the luminance difference value obtained from the rendering results of each of the first point and the second point and the error obtained from the luminance difference value based on either the calculation method of the event threshold included in the actual machine information, the brightening threshold and darkening threshold of the event, or the luminance difference between the luminance value when the previous event was output at the same pixel and the luminance value when the next event is output.

17. The image processing device according to claim 9, wherein the acquisition unit acquires the actual machine information through an input operation by a user.

18. An image processing method comprising: a computer acquiring event data indicating the content of an event detected by an image sensor and actual machine information, which is information regarding the settings when the event data was obtained; and constructing a model based on the actual machine information that can generate an image corresponding to an arbitrary camera position when the camera position is given, wherein the actual machine information includes a method for calculating an event threshold used when the image sensor detects the event.

Images