Event data output sensor and electronic device

Abstract

The present disclosure relates to an event data output sensor and an electronic device that make it possible to switch the output of output data of different output formats in a plurality of switching units. This event data output sensor comprises: a data conversion processing unit that converts event data indicating the content of an event detected on the basis of a change in a luminance signal indicating the luminance of light received by a plurality of event pixels into output data of different output formats; and a selection processing unit that selects, in a predetermined switching unit, output data of any one output format from among the output data of a plurality of different output formats, and outputs the selected output data. The selection processing unit switches the output of the output data in units of frames, areas or lines. The present technology can be applied to, for example, an EVS that detects event data.

Description

Event data output sensors and electronic devices

[0001] The present disclosure relates to an event data output sensor and an electronic device, and more particularly to an event data output sensor and an electronic device that are capable of switching between outputting output data with different output formats in multiple switching units.

[0002] In recent years, development has been progressing on an event-based vision sensor (EVS) that detects a change in luminance of each pixel as an event in real time and outputs event data based on the occurrence of the event.

[0003] Furthermore, Patent Document 1 discloses a sensor architecture that uses a hybrid frame-based and event-based method.

[0004] Special Publication No. 2017-535999

[0005] Incidentally, conventional EVSs are configured to be able to output output data of different output formats in which event data is compressed using a plurality of different encoding methods, for example, but it is not possible to switch between these output data of different output formats in multiple switching units.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to switch between output data in different output formats in multiple switching units.

[0007] An event data output sensor according to one aspect of the present disclosure includes a data conversion processing unit that converts event data indicating the content of an event detected based on changes in a luminance signal indicating the luminance of light received by a plurality of event pixels into output data with different output formats, and a selection processing unit that selects and outputs the output data in one of the plurality of different output formats at a predetermined switching unit.

[0008] An electronic device according to one aspect of the present disclosure includes an event data output sensor having a data conversion processing unit that converts event data indicating the content of an event detected based on changes in a luminance signal indicating the luminance of light received by a plurality of event pixels into output data with different output formats, and a selection processing unit that selects and outputs the output data in one of the plurality of different output formats at a predetermined switching unit.

[0009] In one aspect of the present disclosure, event data indicating the content of an event detected based on changes in a luminance signal indicating the brightness of light received by multiple event pixels is converted into output data of different output formats, and output data of any one of the multiple different output formats is selected at a predetermined switching unit and output.

[0010] 1 is a block diagram showing a configuration example of a first embodiment of an EVS system to which the present technology is applied. FIG. 2 is a block diagram illustrating a first variation of an EVS. FIG. 3 is a diagram illustrating an example of a frame structure. FIG. 4 is a block diagram illustrating a second variation of an EVS. FIG. 5 is a diagram illustrating information described in a line header. FIG. 6 is a diagram illustrating information described in a frame header or a line header. FIG. 7 is a block diagram illustrating a third variation of an EVS. FIG. 8 is a diagram illustrating an example of a frame structure. FIG. 9 is a block diagram showing a configuration example of a second embodiment of an EVS system to which the present technology is applied. FIG. 10 is a block diagram showing a configuration example of a third embodiment of an EVS system to which the present technology is applied. FIG. 11 is a block diagram showing a configuration example of a fourth embodiment of an EVS system to which the present technology is applied. FIG. 12 is a block diagram showing a configuration example of a fifth embodiment of an EVS system to which the present technology is applied. FIG. 13 is a block diagram showing a configuration example of a sixth embodiment of an EVS system to which the present technology is applied. FIG. 14 is a block diagram showing a configuration example of a seventh embodiment of an EVS system to which the present technology is applied. FIG. 15 is a block diagram showing a configuration example of an eighth embodiment of an EVS system to which the present technology is applied. FIG. 16 is a diagram illustrating input data. FIG. 17 is a diagram illustrating output data of Hcomp coding. FIG. 18 is a diagram illustrating output data of Run Length coding. FIG. 19 is a diagram illustrating output data of Huffman coding. FIG. 19 is a diagram illustrating output data of Light Huffman coding. FIG. 10 is a diagram illustrating output data of Event Distance coding. FIG. 11 is a diagram illustrating output data from which optical flow has been generated. FIG. 12 is a diagram illustrating output data from which feature points have been extracted. FIG. 13 is a diagram illustrating output data from which an ROI has been extracted. FIG. 14 is a diagram illustrating an example of an ROI cut out from a frame. FIG. 15 is a diagram illustrating a first example of an output format. FIG. 16 is a diagram illustrating a second example of an output format. FIG. 17 is a diagram illustrating a third example of an output format. FIG. 18 is a diagram illustrating a fourth example of an output format. FIG. 19 is a diagram illustrating a fifth example of an output format. FIG. 19 is a diagram illustrating an example of use in which an image sensor is used.

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <First Configuration Example of EVS System> FIG. 1 is a block diagram showing a configuration example of a first embodiment of an EVS system to which the present technology is applied.

[0013] 1 is configured by connecting an EVS 12 and an application processor 13 in accordance with a communication standard such as CSI-2 (Camera Serial Interface-2), I2C (Inter-Integrated Circuit), or I3C (Improved Inter Integrated Circuits). The physical layer of the communication can use D-PHY, an interface standard for mobile devices, or A-PHY, an interface standard for in-vehicle devices.

[0014] The EVS 12 includes an EVS pixel array unit 21 , an event signal processing circuit 22 , and an output I / F (interface) 23 .

[0015] The EVS pixel array unit 21 is configured by arranging multiple EVS pixels in an array in a pixel region. Each EVS pixel detects an event based on a change in the luminance value of the received light and outputs an event signal indicating the polarity of the event. For example, an EVS pixel outputs an event signal indicating a positive event when the luminance value changes from a reference value to the positive side, and outputs an event signal indicating a negative event when the luminance value changes from the reference value to the negative side. The event signal for each EVS pixel is then supplied from the EVS pixel array unit 21 to the event signal processing circuit 22.

[0016] The event signal processing circuit 22 performs signal processing on the event signal supplied from the EVS pixel array unit 21 to obtain event data, which is then converted into output data in different output formats and output. As shown in the figure, the event signal processing circuit 22 has a signal processing unit 31, a data conversion processing unit 32, and a selection processing unit 33.

[0017] The signal processing unit 31 performs signal processing to generate event data from the event signal for each EVS pixel, and supplies the event data to the data conversion processing unit 32. For example, the event data includes an address x indicating the position in the x direction on the EVS pixel array unit 21 of the EVS pixel where the event was detected, an address y indicating the position in the y direction on the EVS pixel array unit 21 of the EVS pixel where the event was detected, polarity information p indicating the polarity of the event, and a timestamp t indicating the time when the event was detected.

[0018] The data conversion processing unit 32 has a plurality of data conversion units 41 (N data conversion units 41-1 to 41-N in the illustrated example). The data conversion processing unit 32 converts the event data supplied from the signal processing unit 31 into different output formats in the data conversion units 41, and supplies the output data in the different output formats to the selection processing unit 33. For example, the data conversion processing unit 32 outputs, as output data, uncompressed event data, compressed data obtained by encoding the event data using a plurality of different encoding methods, metadata such as optical flow, feature points, and ROI generated from the event data, or representation data (e.g., event count) obtained by converting the event data to represent it in a different way.

[0019] The selection processing unit 33 selects output data of one output format from among a plurality of output data of different output formats supplied from the data conversion processing unit 32 in various switching units (for example, frame units, area units, or line units), and supplies the selected output data to the output I / F 23. The selection of output data by the selection processing unit 33 will be described later with reference to Figures 9 to 15.

[0020] The output I / F 23 transmits the output data supplied from the selection processing unit 33 of the event signal processing circuit 22 to the application processor 13 in accordance with a communication standard such as CSI-2.

[0021] The application processor 13 performs various data processing based on the output data supplied from the EVS 12, and executes various applications according to the event data.

[0022] The EVS system 11 is configured as described above, and can switch between output data with different output formats, for example, on a frame-by-frame, area-by-area, or line-by-line basis, and supply event data from the EVS 12 to the application processor 13. In this way, the EVS 12 can switch between output data with different output formats in desired switching units to widen the output variation, thereby enabling the EVS system 11 to reduce the processing load of subsequent processing, for example. In other words, if the compression method differs for each line, the load of the decompression process may increase, and the EVS system 11 can avoid such an increase in the load of the decompression process.

[0023] Furthermore, the EVS system 11 can accommodate a wide range of system optimizations, such as maximizing the efficiency of the amount of output data from the EVS 12. This can improve the usability of the EVS system 11.

[0024] <Variations of Data Conversion Processing Unit> Variations of the data conversion processing unit 32 will be described with reference to FIGS.

[0025] Fig. 2 is a diagram illustrating a data conversion processing unit 32a, which is a first variation. In the event signal processing circuit 22a shown in Fig. 2, blocks common to the event signal processing circuit 22a shown in Fig. 1 are assigned the same reference numerals, and detailed descriptions thereof will be omitted.

[0026] 2, the event signal processing circuit 22a includes a signal processing unit 31, a data conversion processing unit 32a, and a selection processing unit 33a. The data conversion processing unit 32a includes a decompression processing unit 51, a first compression processing unit 52, a second compression processing unit 53, a metadata generation processing unit 54, and a representation processing unit 55.

[0027] The decompression processing unit 51 outputs the event data supplied from the signal processing unit 31 as output data without modification. For example, the decompression processing unit 51 is configured with a buffer or the like, and the event data output from the decompression processing unit 51 (i.e., the original data supplied from the signal processing unit 31) is referred to as uncompressed event data.

[0028] The first compression processing unit 52 performs a first compression process to encode the event data supplied from the signal processing unit 31 using a first encoding method, and outputs the first compressed data, which is the event data encoded using the first encoding method, as output data.

[0029] The second compression processing unit 53 performs a second compression process to encode the event data supplied from the signal processing unit 31 using a second encoding method different from the first encoding method, and outputs the second compressed data, which is the event data encoded using the second encoding method, as output data.

[0030] The metadata generation processing unit 54 performs a metadata generation process to generate metadata (e.g., optical flow, feature points, ROI, etc.) from the event data supplied from the signal processing unit 31, and outputs the metadata generated from the event data as output data.

[0031] The representation processing unit 55 performs representation processing to convert the event data supplied from the signal processing unit 31 into representation data, and outputs the representation data converted from the event data as output data.

[0032] For example, the representation data can be obtained by converting the event data so that, for each EVS pixel, the representation data is expressed as 1 if an event is detected one or more times within a certain period, and 0 if no event is detected within the certain period. The representation data can also be obtained by converting the event data so that, for each EVS pixel, the representation data is expressed as an expression indicating the number of times an event is detected within a certain period (i.e., an event count). The representation data can also be obtained by converting the event data so that, for each frame, the time starts at zero, and the representation data is expressed as an expression indicating the most recent time value at which an event was detected within the frame. The representation data can also be obtained by converting the event data so that, for each frame, the time starts at zero, and the representation data is expressed as an expression indicating a predetermined number of time values ​​counting from the most recent event detected within the frame. The representation data can also be obtained by calculating, for each EVS pixel, a normalized value of the time at which an event was detected, and converting the event data so that the representation data is expressed as an expression indicating the sum of values ​​obtained by passing the difference between that value and the current time through a clip function.

[0033] Therefore, the data conversion processing unit 32a supplies the uncompressed event data, the first compressed data, the second compressed data, the metadata, and the representation data to the selection processing unit 33a as output data in different output formats.

[0034] The selection processing unit 33a selects one output data from the uncompressed event data, the first compressed data, the second compressed data, the metadata, and the representation data for each frame corresponding to one screen of the EVS pixel array unit 21, and supplies the selected output data to the output I / F 23. In the example shown in FIG. 2, the selection processing unit 33a selects the first compressed data in frame F-1, selects the uncompressed event data in frame F, and selects the second compressed data in frame F+1. Note that the selection processing unit 33a can switch the selection of output data every arbitrary number of frames N, and the number of frames N does not need to be constant.

[0035] The event signal processing circuit 22a having such a data conversion processing unit 32a can describe the output format of the output data stored in each frame in the frame header FH provided for each frame.

[0036] Fig. 3A shows an example of a frame structure in which uncompressed event data, metadata, or representation data is stored in the payload of each packet. Fig. 3B shows an example of a frame structure in which first compressed data or second compressed data is stored in the payload of each packet, and the data length of each packet varies (expands or contracts) depending on the compression rate of the event data.

[0037] 3, one frame is made up of multiple packets from a frame start FS to a frame end EF, with a packet header PH at the beginning of each packet and a packet footer PF at the end of each packet. The frame header FH, which follows the frame start FS, can describe the output format of the output data stored in each frame.

[0038] Fig. 4 is a diagram illustrating a data conversion processing unit 32b, which is a second variation. In the event signal processing circuit 22b shown in Fig. 4, blocks common to the event signal processing circuit 22 shown in Fig. 1 and the event signal processing circuit 22a shown in Fig. 2 are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0039] 4, the event signal processing circuit 22b is configured to include a signal processing unit 31, a data conversion processing unit 32b, and a selection processing unit 33b. Similar to the data conversion processing unit 32a in FIG. 2, the data conversion processing unit 32b includes a decompression processing unit 51, a first compression processing unit 52, a second compression processing unit 53, a metadata generation processing unit 54, and a representation processing unit 55. Therefore, the data conversion processing unit 32b supplies the decompressed event data, the first compressed data, the second compressed data, the metadata, and the representation data to the selection processing unit 33b as output data.

[0040] The selection processing unit 33b selects one output data from the uncompressed event data, the first compressed data, the second compressed data, the metadata, and the representation data in area units, which are predetermined parts of a frame corresponding to one screen of the EVS pixel array unit 21, and supplies the selected output data to the output I / F 23. For example, the selection processing unit 33b can arbitrarily set areas within a frame, and the number of areas set within a frame can also be arbitrarily set.

[0041] For example, the selection processing unit 33b can switch the output data for various areas (lengths) on each line across an entire frame, or the selection processing unit 33b can switch the output data for a desired partial area of ​​a frame (a part of the lower right in the illustrated example).

[0042] The event signal processing circuit 22b equipped with such a data conversion processing unit 32b can describe the output format and placement position of the output data stored in each area in the frame header FH provided for each frame or the line header LH provided for each line.

[0043] For example, when output data is switched in various areas (lengths) on each line throughout one frame, the line header LH describes the output format and location of the output data stored on that line. That is, as shown in Fig. 5, a case will be described in which second compressed data, first compressed data, uncompressed event data, and representation data are switched on one line. In this case, the line header LH describes that second compressed data is stored and the location of the second compressed data, that first compressed data is stored and the location of the first compressed data, that uncompressed event data is stored and the location of the uncompressed event data, and that representation data is stored and the location of the representation data.

[0044] As shown in A of Fig. 6, when output data is switched in a predetermined partial area within one frame (in the illustrated example, a part of the lower right), the frame header FH shown in B of Fig. 6 describes the output format and coordinates of the output data stored in that frame. That is, the frame header FH describes that uncompressed event data and first compressed data are stored, and describes the coordinates of the four corners of the first compressed data as shown in A of Fig. 6. Note that when output data stored in one frame has two output formats, it is not necessary to describe the output format of the output data stored in the frame header FH.

[0045] Alternatively, as shown in A of Fig. 6, even when output data is switched in a predetermined partial area of ​​one frame (in the illustrated example, a part of the lower right), the output format and layout position of the output data stored in each line may be described in the same manner as the line header LH shown in Fig. 5. That is, as shown in B of Fig. 6, when first compressed data is stored at the right end of the bottom three lines, the line header LH of those three lines will state that first compressed data is stored and the layout position of the first compressed data.

[0046] Fig. 7 is a diagram illustrating a data conversion processing unit 32c, which is a third variation. In the event signal processing circuit 22c shown in Fig. 7, blocks common to the event signal processing circuit 22 shown in Fig. 1 and the event signal processing circuit 22a shown in Fig. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0047] 7, the event signal processing circuit 22c includes a signal processing unit 31, a data conversion processing unit 32c, and a selection processing unit 33c. The data conversion processing unit 32c includes representation processing units 55A to 55D.

[0048] The representation processing units 55A to 55D convert the event data into representation data A to D of different output formats (types), and output the data.

[0049] The selection processing unit 33c selects one of the representation data A to D as output data for each line in the row direction of the EVS pixel array unit 21, and supplies the selected data to the output I / F 23. In the example shown in Fig. 7, the selection processing unit 33c selects representation data A for the first line, representation data C for the second line, representation data B for the third line, representation data A for the fourth line, and representation data B for the fifth line. Note that the selection processing unit 33c can switch the selection of output data every arbitrary number of lines N, and the number of lines N does not need to be constant.

[0050] The event signal processing circuit 22C having such a data conversion processing unit 32C can describe the output format of the output data stored in each line in the line header LH provided for each line.

[0051] FIG. 8 shows an example of a frame structure in which representation data for each line is stored in the payload of each packet.

[0052] 8, one frame is made up of multiple packets from a frame start FS to a frame end EF, with a packet header PH at the beginning of each packet and a packet footer PF at the end of each packet. The line header LH, placed next to the packet header PH, describes the output format of the output data stored in each line.

[0053] Of course, similar to the configuration in which representation data of different output formats is selected for each line, the configuration may also be such that uncompressed event data, first compressed data, second compressed data, metadata, or representation data is selected for each line.

[0054] <Second to Eighth Configuration Examples of EVS Systems> Configuration examples of second to eighth embodiments of EVS systems will be described with reference to Figures 9 to 15. Note that in EVS systems 11A to 11G shown in Figures 9 to 15, blocks that are common to the EVS system 11 in Figure 1 are given the same reference numerals, and detailed descriptions thereof will be omitted.

[0055] FIG. 9 is a block diagram showing a configuration example of the second embodiment of the EVS system.

[0056] 9, in the EVS system 11A, the EVS 12A has the same configuration as the EVS 12 in Fig. 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, and differs from the EVS 12 in that it includes an event signal processing circuit 22A and a register 24. The event signal processing circuit 22A has the same configuration as the event signal processing circuit 22 in Fig. 1 in that it includes a signal processing unit 31 and a data conversion processing unit 32, and differs from the event signal processing circuit 22 in that it includes a selection processing unit 33A.

[0057] The application processor 13A sets designation information that designates the output data to be selected by the selection processing unit 33A in the register 24. The EVS system 11A is provided with a signal line through which the application processor 13A sets the designation information in the register 24, and a signal line through which the selection processing unit 33A reads the designation information from the register 24.

[0058] The selection processing unit 33 A selects one of the plurality of output data supplied from the data conversion processing unit 32 in accordance with the designation information read from the register 24 , and supplies the selected output data to the output I / F 23 .

[0059] For example, the application processor 13A performs neural network processing or the like on the event data detected by the EVS 12A and determines whether to switch output data based on the number of events, reliability, etc. The application processor 13A may also determine whether to switch output data based on the scene, processing, etc. When the application processor 13A determines to switch output data, it can write designation information that specifies the output data to be selected by the selection processing unit 33A to the register 24 via communication using I2C or I3C.

[0060] The EVS system 11A is configured in this manner, and output data selected in accordance with feedback from the application processor 13A is transmitted from the EVS 12A to the application processor 13A. This allows the EVS system 11A to switch between output data of different output formats in desired switching units and supply event data from the EVS 12A to the application processor 13A.

[0061] FIG. 10 is a block diagram showing a configuration example of the third embodiment of the EVS system.

[0062] 10, in an EVS system 11B, an EVS 12B has the same configuration as the EVS 12 in Fig. 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, and differs from the EVS 12 in that it includes an event signal processing circuit 22B. The event signal processing circuit 22B has the same configuration as the event signal processing circuit 22 in Fig. 1 in that it includes a signal processing unit 31 and a data conversion processing unit 32, and differs from the event signal processing circuit 22 in that it includes a selection processing unit 33B.

[0063] The selection processing unit 33B has a data amount calculation unit 61 that calculates the data amount of each of the plurality of output data supplied from the data conversion processing unit 32. Then, the selection processing unit 33B selects one of the plurality of output data supplied from the data conversion processing unit 32D in accordance with the data amount calculated by the data amount calculation unit 61, and supplies the selected output data to the output I / F 23. For example, the selection processing unit 33B can select the output data with the smallest data amount for each switching unit.

[0064] The EVS system 11B is configured in this manner, and output data selected in accordance with the data amounts of the plurality of output data output from the data conversion processing unit 32 is transmitted from the EVS 12B to the application processor 13. This allows the EVS system 11B to switch between output data of different output formats in desired switching units and supply event data from the EVS 12B to the application processor 13.

[0065] FIG. 11 is a block diagram showing a configuration example of the fourth embodiment of the EVS system.

[0066] 11 , in an EVS system 11C, an EVS 12C is configured similarly to the EVS 12 of FIG. 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, but differs from the EVS 12 of FIG. 1 in that it includes an event signal processing circuit 22C. The event signal processing circuit 22C is configured similarly to the event signal processing circuit 22 of FIG. 1 in that it includes a signal processing unit 31 and a data conversion processing unit 32, but differs from the event signal processing circuit 22 of FIG. 1 in that it includes a selection processing unit 33C. The event signal processing circuit 22C is configured so that event data output from the signal processing unit 31 is supplied to the selection processing unit 33C, and the event data is supplied from the selection processing unit 33C to one of a plurality of data conversion units 41 included in the data conversion processing unit 32.

[0067] The selection processing unit 33C has a scene determination unit 62 that determines a scene (which data conversion is appropriate) based on the event data supplied from the signal processing unit 31. For example, the scene determination unit 62 can determine a scene for each switching unit according to an algorithm such as event number analysis, firing rate analysis, cluster analysis, or determination AI (Artificial Intelligence).

[0068] The selection processing unit 33C then selects one of the multiple data conversion units 41 included in the data conversion processing unit 32 in accordance with the scene determined by the scene determination unit 62, and supplies the event data to that data conversion unit 41. Therefore, in the event signal processing circuit 22C, only the one data conversion unit 41 to which the event data is supplied is driven, thereby achieving further reduction in power consumption.

[0069] The EVS system 11C is configured in this way, and event data is supplied to a data conversion unit 41 selected in accordance with a scene determined based on the event data output from the signal processing unit 31, and the output data converted by the data conversion unit 41 is transmitted from the EVS 12C to the application processor 13. In this way, the EVS system 11C can switch between output data of different output formats in desired switching units and supply the event data from the EVS 12C to the application processor 13.

[0070] FIG. 12 is a block diagram showing a configuration example of the fifth embodiment of the EVS system.

[0071] 12, in an EVS system 11D, an EVS 12D has the same configuration as the EVS 12 in Fig. 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, and differs from the EVS 12 in Fig. 1 in that it includes an event signal processing circuit 22D. The event signal processing circuit 22D has the same configuration as the event signal processing circuit 22 in Fig. 1 in that it includes a signal processing unit 31, and differs from the event signal processing circuit 22 in Fig. 1 in that it includes a data conversion processing unit 32D, a selection processing unit 33D, and a comparison unit 34.

[0072] 1, the data conversion processing unit 32D has a plurality of data conversion units 41D (N data conversion units 41D-1 to 41D-N in the illustrated example). The data conversion unit 41D supplies output data to the selection processing unit 33D, and also supplies the data amount of the output data to the comparison unit 34 for each switching unit.

[0073] The comparison unit 34 compares the data amount of each of the plurality of output data for each switching unit, and notifies the selection processing unit 33D of the comparison result.

[0074] The selection processing unit 33D selects one of the plurality of output data supplied from the data conversion processing unit 32D in accordance with the comparison result supplied from the comparison unit 34, and supplies the selected output data to the output I / F 23.

[0075] The EVS system 11D is configured in this manner, and output data selected in accordance with the comparison result of the data amounts of the plurality of output data output from the data conversion processing unit 32D is transmitted from the EVS 12D to the application processor 13. As a result, the EVS system 11D can switch between output data of different output formats in desired switching units and supply event data from the EVS 12D to the application processor 13.

[0076] FIG. 13 is a block diagram showing a configuration example of the sixth embodiment of the EVS system.

[0077] 13, in an EVS system 11E, an EVS 12E has the same configuration as the EVS 12 in Fig. 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, and differs from the EVS 12 in that it includes an event signal processing circuit 22E. The event signal processing circuit 22E has the same configuration as the event signal processing circuit 22 in Fig. 1 in that it includes a signal processing unit 31 and a data conversion processing unit 32, and differs from the event signal processing circuit 22 in that it includes a selection processing unit 33E, a memory 35, a data amount calculation unit 36, and a selection processing unit 37.

[0078] In the event signal processing circuit 22E, the signal processing unit 31 supplies the event data to the memory 35 and the data amount calculation unit 36.

[0079] The memory 35 temporarily stores the event data for the processing unit while the data amount calculation unit 36 ​​is calculating the amount of output data.

[0080] The data amount calculation unit 36 ​​calculates the amount of event data for each processing unit supplied from the event signal processing circuit 22E, and supplies a selection signal for selecting a data conversion unit 41 in accordance with the calculated data amount to the selection processing unit 37 and the selection processing unit 33E via a signal line. For example, the data amount calculation unit 36 ​​can select a data conversion unit 41 that can convert the event data so that the amount of output data is minimized.

[0081] The selection processing unit 37 reads out the event data from the memory 35 and supplies it to the data conversion unit 41 in accordance with the selection signal supplied from the data amount calculation unit 36 ​​via a signal line.

[0082] The selection processing unit 33E selects the output data supplied from the data conversion unit 41 in accordance with a selection signal supplied from the data amount calculation unit 36 ​​via a signal line, and supplies the selected data to the output I / F 23.

[0083] The EVS system 11E is configured in this way, and by calculating the data amount of the event data using the data amount calculation unit 36, output data selected to ensure the minimum data amount is transmitted from the EVS 12E to the application processor 13. In this way, the EVS system 11E can switch between output data of different output formats in desired switching units and supply event data from the EVS 12E to the application processor 13.

[0084] It is also possible to configure the system without providing the memory 35, so that the calculation result of the amount of event data for a certain processing unit is used to select the data converter 41 that will convert the event data for the next processing unit.

[0085] FIG. 14 is a block diagram showing a configuration example of the seventh embodiment of the EVS system.

[0086] 14, in an EVS system 11F, an EVS 12F has the same configuration as the EVS 12 in Fig. 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, and differs from the EVS 12 in that it includes an event signal processing circuit 22F. The event signal processing circuit 22F has the same configuration as the event signal processing circuit 22 in Fig. 1 in that it includes a signal processing unit 31 and a data conversion processing unit 32, and differs from the event signal processing circuit 22 in that it includes a selection processing unit 33F, a memory 35, an event counter 38, and a selection processing unit 37F.

[0087] In the event signal processing circuit 22F, the signal processing unit 31 supplies event data to the memory 35 and the event counter 38.

[0088] The memory 35 temporarily stores event data for a processing unit while the event counter 38 counts and analyzes the events.

[0089] The event counter 38 counts the event data for a processing unit supplied from the event signal processing circuit 22E, and supplies a selection signal for selecting a data converter 41 to the selection processing unit 37F and the selection processing unit 33F via a signal line in accordance with the results of various analyses. In accordance with the selection signal, the selection processing unit 37F supplies the event data to the data converter 41, and the selection processing unit 33F supplies the output data supplied from the data converter 41 to the output I / F 23. For example, the event counter 38 can select the data converter 41 that can convert the event data that is predicted to have the smallest amount of output data in accordance with the analysis results of counting the event data.

[0090] For example, the event counter 38 analyzes whether the number of events is large or small, or whether the event distribution is dense or sparse, and selects an appropriate compression method to select the data conversion unit 41. Furthermore, if there are an extremely large number of events, the event counter 38 selects the non-compression processing unit 51. For example, if the event distribution is dense, it is preferable to perform compression processing in groups (multiple events), and if the event distribution is sparse, it is preferable to perform compression processing on an event-by-event basis.

[0091] Furthermore, if the difference in the number of events from the previous frame is large, the event counter 38 determines that there is a high possibility that some kind of change has occurred, and by selecting the metadata generation processing unit 54 or the representation processing unit 55, it is possible to respond to rapid changes.

[0092] The EVS system 11F is configured in this manner, and more appropriate output data is transmitted from the EVS 12F to the application processor 13 in accordance with the analysis result of counting the event data by the event counter 38. As a result, the EVS system 11F can switch between output data of different output formats in desired switching units and supply the event data from the EVS 12F to the application processor 13.

[0093] It is also possible to configure the system without providing the memory 35, so that the analysis result of counting the event data for a certain processing unit is used to select the data converter 41 that will convert the event data for the next processing unit.

[0094] FIG. 15 is a block diagram showing a configuration example of an eighth embodiment of the EVS system.

[0095] As shown in FIG. 15, the EVS system 11G includes an EVS 12G, an application processor 13, and a CMOS image sensor 14.

[0096] 1 in that it includes an EVS pixel array unit 21 and an output I / F 23, and differs from the EVS 12 in that it includes an event signal processing circuit 22G. The event signal processing circuit 22G is configured similarly to the event signal processing circuit 22 in Fig. 1 in that it includes a signal processing unit 31 and a data conversion processing unit 32, and differs from the event signal processing circuit 22 in that it includes a selection processing unit 33G and a selection processing unit 37G.

[0097] The selection processing unit 37G supplies the event data supplied from the signal processing unit 31 to the data conversion unit 41 in accordance with a selection signal supplied from the CMOS image sensor 14 via a signal line.

[0098] The selection processing unit 33 G selects the output data supplied from the data conversion unit 41 in accordance with a selection signal supplied from the CMOS image sensor 14 via a signal line, and supplies the selected data to the output I / F 23 .

[0099] The CMOS image sensor 14 is configured with an RGB pixel array section 71, a signal processing section 72, an image signal processing circuit 73, and an output I / F 74, and the image signal processing circuit 73 has a motion detection section 81, a face detection section 82, a flicker detection section 83, and an pupil detection section 84.

[0100] The RGB pixel array unit 71 has RGB pixels arranged in an array, each outputting a normal RGB pixel signal, and supplies the RGB pixel signal to a signal processing unit 72. The signal processing unit 72 performs signal processing on the RGB pixel signal supplied from the RGB pixel array unit 71 to obtain image data, and supplies the image data to an image signal processing circuit 73.

[0101] The image signal processing circuit 73 performs motion detection processing by a motion detection section 81, face detection processing by a face detection section 82, flicker detection processing by a flicker detection section 83, and pupil detection processing by a pupil detection section 84 on the image data supplied from the signal processing section 72, and supplies the image data to which the results of these processes have been added as metadata to the output I / F 74. The output I / F 74 outputs the image data supplied from the image signal processing circuit 73 to a downstream image processing device (not shown).

[0102] Then, the image signal processing circuit 73 supplies a selection signal to the selection processing unit 33G and the selection processing unit 37G based on the processing results of the motion detection processing by the motion detection unit 81, the face detection processing by the face detection unit 82, the flicker detection processing by the flicker detection unit 83, and the pupil detection processing by the pupil detection unit 84.

[0103] For example, the results of the motion detection process by the motion detection unit 81 tend to be such that when there is a lot of motion, the number of events is large, and when there is little motion, the number of events is small. In accordance with the results of such motion detection process, the image signal processing circuit 73 can select a data conversion unit 41 that reduces the amount of data in the output data.

[0104] Furthermore, if flicker is detected as a result of the flicker detection process by the flicker detection unit 83, the number of events will be extremely high, so the signal-to-noise ratio of the amount of information can be improved by selecting the non-compression processing unit 51 or the representation processing unit 55.

[0105] Furthermore, after a face or an eye is detected as a result of the face detection process by the face detection unit 82 or the eye detection process by the eye detection unit 84, the face or eye detection may be continued at high speed by signal processing within the EVS 12G. Alternatively, after a face or an eye is detected, the application processor 13 may perform post-processing on the output data output from the representation processing unit 55, thereby continuing to detect the face or eye at high speed.

[0106] The EVS system 11G is configured in this manner, and in accordance with a selection signal supplied from the CMOS image sensor 14 via a signal line, more appropriate output data is transmitted from the EVS 12G to the application processor 13. This allows the EVS system 11G to switch between output data of different output formats in desired switching units and supply event data from the EVS 12G to the application processor 13.

[0107] The EVS12G and the CMOS image sensor 14 may be mounted on separate sensor chips or on a single sensor chip. For example, in a configuration in which the EVS12G and the CMOS image sensor 14 are mounted on a single sensor chip, various configuration examples can be employed, such as a configuration example in which the EVS pixels and the RGB pixels are arranged at a uniform density overall, or a configuration example in which the EVS pixels and the RGB pixels are arranged separately for each position, as disclosed in International Publication No. 2023 / 058670.

[0108] <Input Data and Output Data> Input data input to the data conversion processing unit 32 and output data output from the data conversion processing unit 32 will be described with reference to FIGS.

[0109] FIG. 16 shows an example of a data format used for the input data input to the data conversion processing unit 32.

[0110] 16A, the input data may be in a data format consisting of event data arranged in chronological order according to the timing at which the event was detected. For example, the event data may include addresses x and y of the EVS pixels at which the event was detected, polarity information p indicating the polarity of the event (positive event or negative event), and a timestamp t indicating the time at which the event was detected.

[0111] 16B, the input data may be in the form of a count map, which is formed by mapping the number of events for each EVS pixel detected per unit time (e.g., one frame period) according to the arrangement of the EVS pixels. Note that the number of events may be counted by distinguishing between the occurrence of positive events and the occurrence of negative events, or may be counted without distinguishing between the occurrence of positive events and the occurrence of negative events.

[0112] As shown in Fig. 16C, the input data may use an event frame data format in which pixel values ​​indicating the polarity of an event are arranged according to the arrangement of EVS pixels for each frame period. For example, the input data may be represented by a two-bit pixel value (P = '11') indicating that a positive event has been detected, a two-bit pixel value (N = '10') indicating that a negative event has been detected, or a two-bit pixel value (0 = '00') indicating that no event has been detected. Note that if both a positive event and a negative event are detected by the same EVS pixel in one frame period, the polarity of the last detected event may be used as the pixel value of that EVS pixel.

[0113] 16D, the input data may use an event address data format that includes, for each frame period, the address x and address y of an EVS pixel where an event is detected, and polarity information p (positive event or negative event) indicating the polarity of the event. Note that for EVS pixels where no event is detected in one frame period, the input data contains no data.

[0114] FIG. 17 shows an example of output data when Hcomp coding is used as the encoding method for compressing input data.

[0115] In Hcomp coding, input data is divided into the address x of the EVS pixel where the i-th event occurred. ｉ , and the polarity p of the i-th event ｉ This is a method of encoding into output data expressed as (positive event: 0, negative event: 1).

[0116] For example, as shown in the figure, a case will be described in which one line of input data is coded by Hcomp coding, in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels. In this case, following the address y of that line, output data (x ０, p ０ ) = (0, 0) is output. Similarly, output data (x) = (0, 0) indicating that a negative event is detected at the 11th EVS pixel is output. １ , p １ ) = (11, 1), the output data (x ２ , p ２ ) = (12, 1), the output data (x ３ , p ３ )=(13, 1) is output. When outputting the output data of all lines, the line address y is not required.

[0117] FIG. 18 shows an example of output data when run length coding is used as the encoding method for compressing input data.

[0118] In Run Length Coding, input data is divided into the i-th data type d ｉ (positive event: 1, negative event: 2, no event: 0), and the number of consecutive data c ｉ This is a method of encoding data into output data expressed as

[0119] For example, as shown in the figure, a case will be described in which one line of input data is coded by run length coding, in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels. In this case, the address y of that line is followed by output data (d ０ , c ０ ) = (1, 1) is output. Similarly, output data (d １ , c １ ) = (0, 10), and the output data (d ２ , c ２ ) = (2, 3), output data (d ３ , c３ )=(0, 2) is output. When outputting the output data of all lines, the line address y is not required.

[0120] FIG. 19 shows an example of output data when Huffman coding is used as the encoding method for compressing input data.

[0121] In Huffman coding, input data is divided into the i-th coded data h according to the frequency of the number of events (number of data occurrences). ｉ , and output data represented by a frequency table for decoding. Note that one frequency table is provided for each compression unit.

[0122] For example, as shown in the figure, a case will be described in which one line of input data is coded by Huffman coding, in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels. In this case, the address y of that line is followed by the 0th to 5th coded data (h ０ , h １ , h ２ , h ３ , h ４ , h ５ ) = (7, 2, 0, 0, 0, 6) is output, and the frequency table shown in the figure is output. The frequency table contains the following registered information: the Huffman code indicating that the number of occurrences of negative event (= 2) data is 3 is 0 (= 0), the Huffman code indicating that the number of occurrences of data with 10 consecutive no events is 1 is 10 (= 2), the Huffman code indicating that the number of occurrences of data with two consecutive no events is 1 is 100 (= 6), and the Huffman code indicating that the number of occurrences of data with positive event (= 1) is 111 (= 7). Note that when output data for all lines is output, the line address y is not required.

[0123] FIG. 20 shows an example of output data when Light Huffman coding is used as the encoding method for compressing input data.

[0124] Light Huffman coding is a method of encoding input data into output data in which no event is represented by one bit and the presence of an event is represented by two bits.

[0125] For example, as shown in the figure, all events in the input data are represented by two bits (positive event: 01, negative event: 10, no event: 00). When such input data is encoded using Light Huffman coding, the output data represents events by two bits (positive event: 11, negative event: 10) and no event by one bit (:0).

[0126] FIG. 21 shows an example of output data when Event Distance coding is used as the encoding method for compressing input data.

[0127] Event Distance coding is a method of coding input data based on the distance d from the previous event of the i-th event. ｉ , polarity p of the i-th event ｉ (positive event: 0, negative event: 1), and the number of consecutive i-th data c ｉ This is a method of encoding data into output data expressed as

[0128] For example, as shown in the figure, a case will be described in which one line of input data in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels is coded using Event Distance coding. In this case, following the address y of that line, output data (d ０ , p ０ , c ０,) = (0,0,1) is output. Similarly, output data (d １ , p １ , c １ ,)=(10,1,3) is output. When outputting the output data of all lines, the line address y is not required.

[0129] In addition to the encoding methods described with reference to FIGS. 17 to 21 , the present technology can employ various encoding methods, such as the encoding method according to ISSCC (IEEE International Solid-State Circuits Conference) 2017 4.1 and the encoding method according to ISSCC2020 5.10.

[0130] FIG. 22 shows an example of output data when the metadata generation processing unit 54 generates an optical flow that represents the movement of an object on an image using vectors from input data.

[0131] Optical flow generation is performed by calculating the address x of the i-th block in the x direction from the input data. ｉ , the y-direction address y of the i-th block ｉ , the optical flow in the x direction of the i-th block x_dis ｉ , the optical flow in the y direction of the i-th block, y_dis ｉ , and the acquisition time of the i-th optical flow t ｉ When outputting all blocks, the address x ｉ and address y ｉ is not required, and the acquisition time t ｉ is not necessarily required.

[0132] For example, input data in one of the data formats shown in Fig. 16 is input to the metadata generation processing unit 54. Fig. 22 shows input data (x ｉ , y ｉ , p ｉ , t ｉ) is shown. The metadata generation processing unit 54 reconstructs frame data from input data input during one frame period, calculates optical flow for each block, for example, from frame data of one or more previous frames previously reconstructed and frame data of the current frame reconstructed this time, and generates output data (x ｉ , y ｉ , x_dis ｉ , y_dis ｉ , t ｉ ) is output. For example, the optical flow calculated by the metadata generation processing unit 54 may include the amount of movement and the speed of movement of each block. Note that a block may be one pixel, or may be an area of ​​M×N pixels (both M and N are integers equal to or greater than 1).

[0133] FIG. 23 shows an example of output data when the metadata generation processing unit 54 extracts feature points representing characteristic points on an image from input data.

[0134] Feature point extraction is performed by finding the address x of the i-th feature point in the x direction from the input data. ｉ , the y-direction address of the i-th feature point y ｉ , and the extraction time t of the i-th feature point ｉ The extraction time of the i-th feature point is t ｉ is not necessarily required.

[0135] For example, input data in one of the data formats shown in Fig. 16 is input to the metadata generation processing unit 54. Fig. 23 shows input data (x ｉ , y ｉ , p ｉ , t ｉ ) is shown. The metadata generation processing unit 54 reconstructs frame data from input data input during one frame period, executes processing to extract feature points from the reconstructed frame data, and generates output data (x ｉ , y ｉ , t ｉ ) is output.

[0136] FIG. 24 shows an example of output data when the metadata generation processing unit 54 extracts, from the input data, an ROI (Region Of Interest) that indicates a specific region of interest to be processed on an image.

[0137] The ROI extraction is performed by extracting the ROI coordinate x_sta in the x direction, which indicates the start position of the ROI for the i-th frame, from the input data. ｉ , the ROI coordinate x_end in the x direction indicating the end position of the ROI for the i-th frame ｉ , the ROI coordinate y_sta in the y direction indicating the start position of the ROI for the i-th frame ｉ , y_end, the ROI coordinate in the y direction indicating the end position of the ROI for the i-th frame ｉ , and the extraction time of the ROI for the i-th frame t ｉ The ROI extraction time for the i-th frame is t ｉ is not necessarily required.

[0138] For example, input data in one of the data formats shown in Fig. 16 is input to the metadata generation processing unit 54. Fig. 24 shows input data (x ｉ , y ｉ , p ｉ , t ｉ ) is shown. The metadata generation processing unit 54 reconstructs frame data from input data input during one frame period, executes processing to extract an ROI from the reconstructed frame data, and outputs output data (x_sta ｉ , x_end ｉ , y_sta ｉ , y_end ｉ , t ｉ 25 , the metadata generation processing unit 54 may output image data cut out as an ROI from frame data, in addition to or instead of the output data from which the ROI has been extracted.

[0139] <Output Format> The output format of a frame storing output data transmitted from the EVS 12 will be described with reference to FIGS.

[0140] 26 to 30 show examples of output data output by the output I / F 23 in accordance with MIPI CSI-2. As shown in the figures, one frame's worth of output data is stored in multiple packets, from a frame start FS indicating the start of the frame to a frame end EF indicating the end of the frame. A packet header PH is placed at the beginning of each packet, and a packet footer PF is placed at the end of each packet. The frame length and line length of one frame correspond to, for example, the number of vertical and horizontal EVS pixels arranged in an array in the EVS pixel array unit 21.

[0141] As described above, the amount of output data sent from EVS 12 varies depending on which output data the selection processing unit 33 selects from the multiple output data output from the data conversion processing unit 32.

[0142] Therefore, when the selection processing unit 33 selects uncompressed event data, metadata, or representation data as output data, as shown in Figure 26, EVS12 outputs a frame in an output format in which the output data stored in each packet that makes up one frame has the same data length.

[0143] On the other hand, when the selection processing unit 33 selects compressed data, as shown in Figure 27, EVS 12 outputs a frame in an output format in which the output data stored in each packet that makes up one frame has a different data length.

[0144] Furthermore, the EVS 12 can stop outputting the output data for a line for which no event has been detected.

[0145] Therefore, when the selection processing unit 33 selects uncompressed event data, metadata, or representation data as output data and there is a line where an event is not detected, as shown in Figure 28, EVS12 outputs a frame in an output format that has fewer packets than the output format shown in Figure 26 by the number of lines where an event is not detected.

[0146] Similarly, when the selection processing unit 33 selects compressed data, as shown in Figure 29, the EVS 12 outputs a frame in an output format that has fewer packets than the output format shown in Figure 27 by the number of lines where no events are detected.

[0147] Furthermore, in the EVS 12, when the selection processing unit 33 selects compressed data, the transfer time for one line of data is shortened, so the output interval of the horizontal synchronization signal is made variable depending on the amount of data transferred for each line. Therefore, in this case, as shown in Figure 30, the horizontal synchronization signal is output at the timing when the output of the line footer PF for each line is completed. This makes it possible to shorten the transfer time for each line, and therefore the transfer time for one frame of data.

[0148] EVS12 can insert an identification flag indicating the output format of the output data (uncompressed event data, first compressed data, second compressed data, metadata, or representation data) into the output data, packet header PH, or packet footer PF.

[0149] <Example of Use of Image Sensor> FIG. 31 is a diagram showing an example of use of the image sensor (EVS) described above.

[0150] The image sensor described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0151] ・Devices for taking images for viewing purposes, such as digital cameras and mobile devices with camera functions. ・Devices for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a car for safe driving such as automatic stopping, and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. ・Devices for home appliances such as TVs, refrigerators, and air conditioners that take images of user gestures and operate the device according to those gestures. ・Devices for medical and healthcare purposes, such as endoscopes and devices that take images of blood vessels by receiving infrared light. ・Devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. ・Devices for beauty purposes, such as skin measuring devices that take images of the skin and microscopes that take images of the scalp. ・Devices for sports purposes, such as action cameras and wearable cameras for sports, etc. ・Devices for agricultural purposes, such as cameras to monitor the condition of fields and crops.

[0152] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) An event data output sensor comprising: a data conversion processing unit that converts event data indicating the content of an event detected based on changes in a luminance signal indicating the luminance of light received by a plurality of event pixels into output data with different output formats, and a selection processing unit that selects and outputs the output data in one of the plurality of different output formats in predetermined switching units. (2) The event data output sensor according to (1) above, wherein the selection processing unit selects the output data in a desired output format and switches output of the output data in frame units corresponding to one screen of a pixel region in which the plurality of event pixels are arranged in an array, or in area units that are a predetermined part of a frame corresponding to one screen of the pixel region. (3) The event data output sensor according to (2), wherein the data conversion processing unit has: a decompression processing unit that outputs the event data as is as the output data, a first compression processing unit that performs a first compression processing to encode the event data using a first encoding method and outputs first compressed data as the output data, a second compression processing unit that performs a second compression processing to encode the event data using a second encoding method and outputs second compressed data as the output data, a metadata generation processing unit that performs a metadata generation processing to generate metadata from the event data and outputs the metadata as the output data, and a representation processing unit that performs a representation processing to convert the event data into representation data and outputs the representation data as the output data. (4) The event data output sensor according to (3), wherein the metadata generation processing unit generates optical flow, feature points, or ROI (Region Of Interest) as the metadata.(5) The event data output sensor according to any of (2) to (4) above, wherein, when the selection processing unit switches the output of the output data on a frame-by-frame basis, the output format of the output data stored in each of the frames is described in a frame header arranged next to a frame start indicating the start of the frame. (6) The event data output sensor according to any of (2) to (5) above, wherein, when the selection processing unit switches the output of the output data on an area-by-area basis, the output format and arrangement position of the output data stored for each of the areas is described in a frame header arranged for each of the frames or a line header arranged for each line of the output data. (7) The event data output sensor according to any of (1) to (6) above, wherein the selection processing unit selects the output data of a desired output format on a line-by-line basis in the row direction of a pixel area in which a plurality of the event pixels are arranged in an array, and switches the output of the output data. (8) The event data output sensor according to (7) above, wherein the data conversion processing unit has a plurality of representation processing units that output, as the output data, the representation data obtained by performing representation processing to convert the event data into representation data, and the plurality of representation processing units each output a different type of the representation data. (9) The event data output sensor according to (7) above, wherein, when the selection processing unit switches the output of the output data on a line-by-line basis, a line header provided for each line describes an output format of the output data stored in each of the lines. (10) The event data output sensor according to any of (1) to (9) above, further comprising: a register in which designation information that designates the output data to be selected by the selection processing unit is set by an application processor that processes data based on the output data and executes an application in accordance with the event data, and the selection processing unit switches the output of the output data in accordance with the designation information read from the register.(11) The event data output sensor according to any one of (1) to (10) above, wherein the selection processing unit has a data amount calculation unit that calculates the data amount of each of the plurality of output data supplied from the data conversion processing unit, and switches the output of the output data in accordance with the data amount calculated by the data amount calculation unit. (12) The event data output sensor according to any one of (1) to (11) above, wherein the selection processing unit has a scene determination unit that determines a scene based on the event data, and the output format of the output data into which the event data is converted in the data conversion processing unit is changed in accordance with the scene determined by the scene determination unit. (13) The event data output sensor according to any one of (1) to (12) above, further comprising a comparison unit that compares the data amounts of the output data of different output formats output from the data conversion processing unit, and the selection processing unit switches the output of the output data in accordance with the comparison result by the comparison unit. (14) The event data output sensor according to any one of (1) to (13) above, further comprising a data amount calculation unit that calculates the amount of event data for a processing unit, wherein the output format of the output data into which the event data is converted in the data conversion processing unit is changed according to the data amount. (15) The event data output sensor according to (14) above, further comprising a memory that temporarily stores the event data for a processing unit. (16) The event data output sensor according to any one of (1) to (15) above, further comprising an event counter that counts and analyzes the event data for a processing unit, wherein the output format of the output data into which the event data is converted in the data conversion processing unit is changed according to an analysis result by the event counter. (17) The event data output sensor according to (16) above, further comprising a memory that temporarily stores the event data for a processing unit.(18) The event data output sensor according to any one of (1) to (17) above, wherein the output format of the output data into which the event data is converted is changed in the data conversion processing unit according to a result of signal processing performed on image data acquired from RGB pixel signals output from a plurality of RGB pixels. (19) An electronic device comprising an event data output sensor having: a data conversion processing unit that converts event data indicating details of events detected based on changes in luminance signals indicating the luminance of light received by a plurality of event pixels into output data with different output formats, and a selection processing unit that selects and outputs the output data in one of the plurality of different output formats at a predetermined switching unit.

[0153] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0154] REFERENCE SIGNS LIST 11 EVS system, 12 EVS, 13 application processor, 14 CMOS image sensor, 21 EVS pixel array unit, 22 event signal processing circuit, 23 output I / F, 24 register, 31 signal processing unit, 32 data conversion processing unit, 33 selection processing unit, 34 comparison unit, 35 memory, 36 data amount calculation unit, 37 selection processing unit, 38 event counter, 41 data conversion unit, 51 decompression processing unit, 52 first compression processing unit, 53 second compression processing unit, 54 metadata generation processing unit, 55 representation processing unit, 61 data amount calculation unit, 62 scene determination unit, 71 RGB pixel array unit, 72 signal processing unit, 73 image signal processing circuit, 74 output I / F, 81 motion detection unit, 82 face detection unit, 83 flicker detection unit 84 pupil detection unit

Claims

1. An event data output sensor comprising: a data conversion processing unit that converts event data, which indicates the content of an event detected based on changes in luminance signals that indicate the luminance of light received by multiple event pixels, into output data with different output formats; and a selection processing unit that selects and outputs one of the multiple output data with different output formats at a predetermined switching unit.

2. The event data output sensor according to claim 1, wherein the selection processing unit selects the output data in a desired output format for each frame corresponding to one screen of a pixel region in which a plurality of the event pixels are arranged in an array, or for each area that is a predetermined part of a frame corresponding to one screen of the pixel region, and switches the output of the output data.

3. The event data output sensor according to claim 2, wherein the data conversion processing unit comprises: a decompression processing unit that outputs the event data as is as the output data; a first compression processing unit that performs a first compression processing to encode the event data using a first encoding method and outputs first compressed data as the output data; a second compression processing unit that performs a second compression processing to encode the event data using a second encoding method and outputs second compressed data as the output data; a metadata generation processing unit that performs a metadata generation processing to generate metadata from the event data and outputs the metadata as the output data; and a representation processing unit that performs a representation processing to convert the event data into representation data and outputs the representation data as the output data.

4. The event data output sensor according to claim 3, wherein the metadata generation processing unit generates optical flow, feature points, or ROI (Region Of Interest) as the metadata.

5. The event data output sensor according to claim 2, wherein when the selection processing unit switches the output of the output data on a frame-by-frame basis, the output format of the output data stored in each of the frames is described in a frame header that is placed next to a frame start that indicates the start of the frame.

6. The event data output sensor according to claim 2, wherein when the selection processing unit switches the output of the output data on an area-by-area basis, the output format and arrangement position of the output data stored for each of the areas are described in a frame header provided for each of the frames or a line header provided for each of the lines of the output data.

7. The event data output sensor according to claim 1, wherein the selection processing unit selects the output data in a desired output format for each line in the row direction of a pixel region in which a plurality of the event pixels are arranged in an array, and switches the output of the output data.

8. The event data output sensor according to claim 7, wherein the data conversion processing unit has a plurality of representation processing units that output the representation data obtained by performing representation processing to convert the event data into representation data as the output data, and the plurality of representation processing units each output a different type of representation data.

9. The event data output sensor according to claim 7, wherein when the selection processing unit switches the output of the output data on a line-by-line basis, the output format of the output data stored in each line is described in a line header provided for each line.

10. The event data output sensor according to claim 1, further comprising: a register in which designation information specifying the output data to be selected by the selection processing unit is set by an application processor that performs data processing based on the output data and executes an application in accordance with the event data; and the selection processing unit switches the output of the output data in accordance with the designation information read from the register.

11. The event data output sensor according to claim 1, wherein the selection processing unit has a data amount calculation unit that calculates the data amount of each of the plurality of output data supplied from the data conversion processing unit, and switches the output of the output data in accordance with the data amount calculated by the data amount calculation unit.

12. The event data output sensor according to claim 1, wherein the selection processing unit has a scene determination unit that determines a scene based on the event data, and the output format of the output data that converts the event data in the data conversion processing unit is changed according to the scene determined by the scene determination unit.

13. The event data output sensor according to claim 1, further comprising a comparison unit that compares the data amounts of the output data in different output formats output from the data conversion processing unit, and the selection processing unit switches the output of the output data according to the comparison result by the comparison unit.

14. The event data output sensor according to claim 1, further comprising a data amount calculation unit that calculates the amount of the event data for a processing unit, and the output format of the output data into which the event data is converted in the data conversion processing unit is changed according to the data amount.

15. The event data output sensor according to claim 14, further comprising a memory for temporarily storing the event data for a processing unit.

16. The event data output sensor according to claim 1, further comprising an event counter that counts and analyzes the event data for a processing unit, and the output format of the output data into which the event data is converted in the data conversion processing unit is changed according to the analysis result by the event counter.

17. The event data output sensor according to claim 16, further comprising a memory for temporarily storing the event data for a processing unit.

18. The event data output sensor according to claim 1, wherein the output format of the output data used to convert the event data in the data conversion processing unit is changed according to the results of signal processing performed on image data obtained from RGB pixel signals output from a plurality of RGB pixels.

19. An electronic device equipped with an event data output sensor having: a data conversion processing unit that converts event data indicating the content of an event detected based on changes in luminance signals indicating the luminance of light received by multiple event pixels into output data of different output formats; and a selection processing unit that selects and outputs one of the multiple output data of different output formats at a predetermined switching unit.

Images