Data processing device, data processing method, and event data output sensor

Abstract

The present disclosure relates to a data processing device and a data processing method whereby it is possible to output event data at a higher compression rate. The present disclosure also relates to an event data output sensor. The occurrence of an event is detected on the basis of change in the brightness of light received by a plurality of event detection pixels, the occurrence density of event data indicating the content of the event is used as a basis to change the expression method of an encoding method for encoding the event data, and the event data is compressed. For example, the absolute and relative values of X-direction addresses of the event data within one line are changed on the basis of the occurrence density of the event data within one line. The present technology can be applied to, for example, an EVS that outputs event data on the basis of the occurrence of an event.

Description

Data processing device, data processing method, and event data output sensor

[0001] The present disclosure relates to a data processing device, a data processing method, and an event data output sensor, and in particular to a data processing device, a data processing method, and an event data output sensor that are capable of outputting event data with a higher compression rate.

[0002] In recent years, development has been progressing on an event-based vision sensor (EVS) that detects a change in luminance of each pixel as an event in real time and outputs event data based on the occurrence of the event.

[0003] In EVS, it is not necessary to output pixel data for all pixels, but it is sufficient to output event data only for pixels where an event has occurred. For this reason, various coding methods (Hcomp coding, Run Length coding, Huffman coding, Light Huffman coding, Event Distance coding, Entropy coding, etc.) have been proposed, as disclosed in, for example, Patent Document 1.

[0004] International Publication No. 2022 / 091826

[0005] However, each of the conventional encoding methods has its own advantages and disadvantages, and there is a demand for an encoding method that can achieve a higher compression ratio.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to output event data with a higher compression rate.

[0007] A data processing device according to one aspect of the present disclosure includes an event detection unit that detects the occurrence of an event based on a change in the brightness of light received by a plurality of event detection pixels and generates event data indicating the content of the event, and a compression unit that compresses the event data by changing the representation method of an encoding method that encodes the event data based on the occurrence density of the event data.

[0008] A data processing method according to one aspect of the present disclosure includes a data processing device detecting the occurrence of an event based on a change in the brightness of light received by a plurality of event detection pixels, generating event data indicating the content of the event, and compressing the event data by changing the representation method of an encoding scheme for encoding the event data based on the occurrence density of the event data.

[0009] An event data output sensor according to one aspect of the present disclosure includes a sensor unit in which a plurality of event detection pixels are arranged in an array in a pixel region, an event detection unit that detects the occurrence of an event based on a change in the luminance of light received by the plurality of event detection pixels and generates event data indicating the content of the event, and a compression unit that compresses the event data by changing the expression method of an encoding method that encodes the event data based on the occurrence density of the event data.

[0010] In one aspect of the present disclosure, the occurrence of an event is detected based on a change in the brightness of light received by a plurality of event detection pixels, event data indicating the content of the event is generated, and the event data is compressed by changing the expression method of the encoding method for encoding the event data based on the occurrence density of the event data.

[0011] 1 is a block diagram illustrating an example configuration of an embodiment of an EVS to which the present technology is applied. FIG. 1 is a diagram illustrating the frame structure of one frame of event data. FIG. 2 is a diagram illustrating an example configuration of an event unit. FIG. 3 is a diagram illustrating data amount reduction using Hybrid Hcomp coding. FIG. 4 is a diagram illustrating an address difference threshold. FIG. 5 is a flowchart illustrating encoding processing using Hybrid Hcomp coding. FIG. 6 is a diagram illustrating an example configuration of compressed data configured in byte units. FIG. 7 is a diagram illustrating a modified example of compressed data configured in byte units. FIG. 8 is a diagram illustrating application of the encoding method of the present technology to addresses in the Y direction. FIG. 9 is a diagram illustrating application of the encoding method of the present technology to timestamps. FIG. 10 is a diagram illustrating input data. FIG. 11 is a diagram illustrating output data of Hcomp coding. FIG. 12 is a diagram illustrating output data of Run Length coding. FIG. 13 is a diagram illustrating output data of Huffman coding. FIG. 14 is a diagram illustrating output data of Light Huffman coding. FIG. 15 is a diagram illustrating output data of Event Distance coding. FIG. 16 is a diagram illustrating output data from which optical flow is generated. FIG. 17 is a diagram illustrating output data from which feature points are extracted. FIG. 18 is a diagram illustrating output data from which an ROI is extracted. FIG. 19 is a diagram illustrating an example of an ROI cut out from a frame. FIG. 20 is a diagram illustrating an example use of an image sensor.

[0012] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0013] <Configuration Example of EVS> An EVS to which the present technology is applied will be described with reference to FIGS. 1 to 5.

[0014] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of an EVS to which the present technology is applied.

[0015] As shown in FIG. 1, the EVS 11 includes a sensor unit 21 , an event detection unit 22 , an event information generation processing unit 23 , a packet generation processing unit 24 , and a data output unit 25 .

[0016] The sensor unit 21 includes a pixel area 31 in which a plurality of event detection pixels are arranged in an array, each of which detects the brightness of received light and outputs a brightness signal indicating the brightness, and a line control unit 32 and a column control unit 33 which control the output of brightness signals from each of the event detection pixels provided in the pixel area 31. Hereinafter, a row of a plurality of event detection pixels arranged horizontally in the pixel area 31 will be referred to as a "line." One frame is made up of event data corresponding to one screen of the pixel area 31.

[0017] The event detection unit 22 detects the occurrence of an event, which is a change in the luminance signal output from the sensor unit 21, generates event data indicating the content of the event (for example, data indicating whether the luminance value has changed to the positive or negative side from the reference value), and supplies the event data to the event information generation processing unit 23 and the packet generation processing unit 24. For example, the event detection unit 22 supplies the event data for each frame to the event / line count unit 41, and supplies the event data for each line to the event count unit 42, the data amount calculation unit 44, the compression unit 45, and the packet generation processing unit 24. Note that the event data generated by the event detection unit 22 is generated when the occurrence of an event is detected, and therefore has the characteristics that the timing of output from the event detection unit 22 is irregular and the data length for each line is not constant.

[0018] The event information generation processing unit 23 generates event information (valid line number information, event number analysis information, line data amount information, and compressed data) that is information related to the event detected by the event detection unit 22, and supplies it to the packet generation processing unit 24. The event information generation processing unit 23 is configured to include an event / line counting unit 41, an event counting unit 42, an event number analysis unit 43, a data amount calculation unit 44, and a compression unit 45.

[0019] The event / line counting unit 41 counts the number of lines containing pixels in which an event has occurred (hereinafter referred to as the number of valid lines) for each frame of event data supplied from the event detection unit 22. The event / line counting unit 41 then supplies valid line number information indicating the number of valid lines for each frame to the packet generation processing unit 24.

[0020] The event counting unit 42 obtains the data length of the event data for each line by counting the number of pixels in which an event has occurred for each line of event data supplied from the event detection unit 22. The event counting unit 42 then supplies data length information indicating the data length of the event data for each line to the event number analysis unit 43.

[0021] The event number analysis unit 43 analyzes whether the data length of the event data for each line is in bytes based on the data length information supplied from the event count unit 42. Then, the event number analysis unit 43 supplies the packet generation processing unit 24 with a signal (Byte_Sig signal / no_Byte_Sig signal) indicating whether the data length of the event data for the line to be processed is in bytes, and event number analysis information including the data length information.

[0022] The data amount calculation unit 44 performs a calculation to calculate the data amount of all the event data, regardless of whether an event has occurred, for each line of event data supplied from the event detection unit 22. Then, the data amount calculation unit 44 supplies line data amount information indicating the data amount for each line to the packet generation processing unit 24 and the compression unit 45.

[0023] The compression unit 45 compresses one line of event data supplied from the event detection unit 22 using a predetermined encoding method (compression technique) and supplies the compressed data obtained by compressing the event data to the packet generation processing unit 24. For example, the compression unit 45 can compress the event data by switching between multiple encoding methods for each line, as shown in Fig. 2B (described later), based on the line data amount information supplied from the data amount calculation unit 44. Furthermore, the compression unit 45 can compress the event data by changing the expression method of the encoding method within one line on an event data unit basis, as shown in Fig. 2C (described later), based on the local occurrence density of event data within each line.

[0024] The packet generation processing unit 24 generates packets in which the event data supplied from the event detection unit 22 is stored for each one or more lines based on the event information supplied from the event information generation processing unit 23 (valid line number information, event number analysis information, line data amount information, and compressed data), and supplies the packets to the data output unit 25. The packet generation processing unit 24 is configured to include a header generation unit 51 and a packet generation unit 52.

[0025] The header generation unit 51 generates various headers to be provided in packets storing event data, based on the event information supplied from the event information generation processing unit 23. For example, as shown in Fig. 2 (described later), the header generation unit 51 generates a packet header PH, which is a header provided at the beginning of each packet, a frame header FH, which is a header provided at the beginning of each frame, and a line header LH, which is a header provided at the beginning of each line.

[0026] For example, the header generation unit 51 sets the number of valid lines to Valid Line Num in the frame header based on the valid line number information. The header generation unit 51 sets the number of lines included in the packet to Line Num in the first line header of the packet, and sets the line number in the packet to Line Num in the second or subsequent line header of the packet. The header generation unit 51 sets 0 to Header Indicator in the frame header and 1 to Header Indicator in the line header. If padding has been inserted, the header generation unit 51 sets 1 to Padding Insertion in the line header. If one line of event data is stored across packets, the header generation unit 51 sets 1 to Data Straddle in the line header.

[0027] The packet generation unit 52 generates a packet that stores compressed data obtained by compressing the event data supplied from the compression unit 45, based on the event information supplied from the event information generation processing unit 23. When the event data is output from the EVS 11 uncompressed, the packet generation unit 52 generates a packet that stores the event data supplied from the event detection unit 22.

[0028] For example, the packet generation unit 52 can generate packets storing multiple lines of event data based on data length information included in the event count analysis information. If the event data is not in bytes, the packet generation unit 52 inserts padding so that the read line length is in bytes based on a signal (Byte_Sig signal / no_Byte_Sig signal) included in the event count analysis information. If the packet generation unit 52 cannot store one line of event data in the previous packet, it stores the remaining portion of the event data that cannot be stored in the previous packet in the next packet. When no event occurs, the packet generation unit 52 generates packets storing dummy data so that the packet output interval is constant. In addition to generating packets storing event data, the packet generation unit 52 may also generate packets storing embedded data.

[0029] The data output unit 25 outputs the packets storing the event data supplied from the packet generation processing unit 24 to an application processor (not shown) in accordance with a communication standard such as CSI-2 (Camera Serial Interface-2) of MIPI (Mobile Industry Processor Interface), I2C (Inter-Integrated Circuit), or I3C (Improved Inter Integrated Circuits). In addition, the physical layer of communication can use D-PHY, which is an interface standard for mobile devices, or A-PHY, which is an interface standard for in-vehicle devices.

[0030] In the EVS 11 configured as described above, the compression unit 45 can compress event data by changing the encoding method for each event data in one line based on the local event data occurrence density within each line. This allows the EVS 11 to output event data with a higher compression rate.

[0031] The frame structure of one frame of event data output from the EVS 11 will be described with reference to FIG.

[0032] As shown in Figure 2, one frame's worth of event data is stored in multiple packets, from a frame start FS indicating the start of the frame to a frame end EF indicating the end of the frame. A packet header PH is placed at the beginning of each packet, and a packet footer PF is placed at the end of each packet. A frame header FH is placed after the frame start FS, and in each packet, a line header LH is placed after the packet header PH (at the beginning of the payload).

[0033] 2A shows the frame structure of one frame in which uncompressed event data that has not been compressed by the compression unit 45 is stored in a packet for each line. For example, the uncompressed event data indicates whether an event has occurred for all pixels on each line, and as shown in the figure, the data length is the same for all lines.

[0034] 2B shows the frame structure of one frame in which event data compressed by the compressor 45 is stored in packets so that multiple encoding methods are switched for each line. In the example shown, the compressor 45 performs encoding by switching between the first encoding method and the second encoding method so that the line length is the shortest, resulting in different data lengths for each line.

[0035] 2C shows the frame structure of one frame in which event data compressed by the compressor 45 is stored in packets so that the encoding method within one line is changed based on the local event data occurrence density within each line. As shown in the figure, the compressor 45 changes the encoding method within each line so that, for event data with a high occurrence density, the event data is encoded using an encoding method (compression (dense)) that provides a high compression rate for high occurrence densities, and for event data with a low occurrence density, the event data is encoded using an encoding method (compression (coarse)) that provides a high compression rate for low occurrence densities. By changing the encoding method based on the local event data occurrence density within a line in this way, the data length can be shortened compared to compressing event data by switching between multiple encoding methods for each line as shown in FIG. 2B.

[0036] For example, Hcomp coding, which is a coding method that achieves the highest compression rate when the occurrence density of event data is low, will be described.

[0037] As shown in Figure 3A, in Hcomp coding, an event unit, which is the information per event, is composed of 12 bits, for example, the absolute value of the X-address (11 bits) and the polarity of the event (1 bit). Note that the y (line) information is not shown.

[0038] However, with Hcomp coding, as the number of events in a line increases and the event data density increases, the X-axis address bits become redundant, resulting in reduced compression efficiency. Specifically, when the horizontal pixel count is 1280 pixels, uncompressed event data represents three states—positive event, negative event, and no event—using two bits. Therefore, regardless of whether an event occurs, the data volume per line is always 2560 bits (=1280 x 2). In contrast, when event data is compressed using Hcomp coding, the data volume is reduced to 360 bits (=30 x 12) when there are 30 events in a line, but increases to 2880 bits (=240 x 12) when there are 240 events in a line.

[0039] Therefore, the compression unit 45 can use Hcomp coding (hereinafter referred to as hybrid Hcomp coding), which converts the X-direction address between absolute and relative values ​​according to the local occurrence density of event data within one line and encodes the absolute and relative X-direction address values ​​within one line. In hybrid Hcomp coding, the absolute X-direction address value is used when the occurrence density of event data is low, and the relative X-direction address value (i.e., the difference from the X-direction address of the immediately preceding event) is used when the occurrence density of event data is high.

[0040] For example, when the occurrence density of event data is low, the event unit is composed of 13 bits, consisting of an indicator (1 bit), an absolute value of the X-direction address (11 bits), and an event polarity (1 bit), as shown in Figure 3B. On the other hand, when the occurrence density of event data is high, the event unit is composed of 5 bits, consisting of an indicator (1 bit), a relative value of the X-direction address (3 bits), and an event polarity (1 bit), as shown in Figure 3C. The indicator is used to identify for each event unit whether an absolute or relative value is used as the X-direction address; if the absolute value of the X-direction address is used, the indicator is set to "0," and if the relative value of the X-direction address is used, the indicator is set to "1."

[0041] For example, as shown in FIG. 4A, the event data will be described in which the polarity of the event at the first pixel in the X-direction address is positive, and the polarity of the events at the 12th, 13th, and 14th pixels in the X-direction address is negative.

[0042] When this event data is compressed using Hcomp coding, the first to fourth event units all become 12 bits, as shown in FIG. 4B, for a total of 48 bits (=12 bits×4).

[0043] On the other hand, when this event data is compressed using Hybrid Hcomp coding, the first and second event units become 13 bits, and the third and fourth event units become 5 bits, as shown in Figure 4C, for a total of 36 bits (= 13 bits × 2 + 5 bits × 2).

[0044] In this way, by the compression unit 45 encoding the event data using Hybrid Hcomp coding, the amount of data per line can be reduced more than when Hcomp coding is used.

[0045] In Hybrid Hcomp coding, for the first event data, the absolute value of the address in the X direction is always used as the base address.

[0046] In Hybrid Hcomp coding, a threshold for determining the occurrence density of event data is set based on the difference between the X-address of the previous event data and the address of the previous event data (hereinafter referred to as the address difference). For example, this difference can be set according to the number of bits used as the relative value of the X-address. If the relative value of the X-address is 3 bits, the threshold for the address difference is set to 7.

[0047] 5, if the X address of the first event data is 7, the compression unit 45 generates an event unit using the absolute value of the X address as is. Next, if the X address of the second event data is 82, the compression unit 45 determines that the address difference from the first event data (75 = 82 - 7) is greater than the threshold value of 7, and generates an event unit using 82, which is the absolute value of the X address.

[0048] Next, when the X address of the third event data is 84, the compression unit 45 determines that the address difference from the second event data (2 = 84 - 82) is equal to or less than the threshold value of 7, and generates an event unit using the relative value of the X address, 2. Similarly, when the X address of the fourth event data is 86, the compression unit 45 determines that the address difference from the third event data (2 = 86 - 84) is equal to or less than the threshold value of 7, and generates an event unit using the relative value of the X address, 2. Then, when the X address of the fifth event data is 100, the compression unit 45 determines that the address difference from the fourth event data (14 = 100 - 86) is greater than the threshold value of 7, and generates an event unit using the absolute value of the X address, 100.

[0049] As described above, Hybrid Hcomp coding is an encoding method that is highly compatible with both sparse and dense event data occurrence densities within a line, and is a more optimal compression method (capable of minimizing data volume) because it can reduce the amount of data per line more than Hcomp coding. Furthermore, Hybrid Hcomp coding compresses event data by changing the encoding method representation within a line (i.e., the absolute or relative value of the X-direction address) for each event data item. This reduces the circuit scale and processing load of the EVS 11 and application processor compared to compression methods that compress event data by switching between multiple encoding methods within a line.

[0050] The encoding method of this embodiment is not limited to the hybrid Hcomp coding, which converts the X-axis address between absolute and relative values ​​according to the local occurrence density of event data within one line, as described above. For example, any encoding method may be used, such that for event data with a high occurrence density within each line, the event data is encoded using an encoding method that provides a high compression rate when the occurrence density is high, and for event data with a low occurrence density, the event data is encoded using an encoding method that provides a high compression rate when the occurrence density is low.

[0051] <Example of Encoding Process Using Hybrid Hcomp Coding> Encoding process using Hybrid Hcomp coding will be described with reference to the flowchart shown in FIG.

[0052] For example, when a line of event data is supplied from the event detection unit 22 to the compression unit 45, the process starts. In step S11, the compression unit 45 acquires the X-direction address of the leading (first) event data.

[0053] In step S12, the compression unit 45 generates an event unit using the absolute value of the X-direction address for the event data whose X-direction address was acquired in step S11.

[0054] In step S13, the compression unit 45 obtains the address in the X direction of the next (second) event data.

[0055] In step S14, the compression unit 45 determines whether the address difference in the X direction address acquired in step S13 is greater than a threshold value.

[0056] If the compression unit 45 determines in step S14 that the address difference is greater than the threshold value, the process proceeds to step S15. In step S15, the compression unit 45 generates an event unit using the absolute value of the X-direction address for the event data whose X-direction address was acquired in step S13, and sets the indicator to "0".

[0057] On the other hand, if the compression unit 45 determines in step S14 that the address difference is not greater than the threshold (i.e., the address difference is equal to or less than the threshold), the process proceeds to step S16. In step S16, the compression unit 45 generates an event unit using the relative value of the X-direction address for the event data whose X-direction address was acquired in step S13, and sets the indicator to "1".

[0058] After the process of step S15 or S16, the process proceeds to step S17, where the compression unit 45 determines whether or not one line of event data has been encoded.

[0059] If the compression unit 45 determines in step S17 that one line of event data has not been encoded, the process returns to step S13. The compression unit 45 then obtains the X-address of the next (second or subsequent) event data, and the same process is repeated thereafter.

[0060] On the other hand, if the compression unit 45 determines in step S17 that one line of event data has been encoded, the process ends.

[0061] As described above, the compression unit 45 encodes the event data using Hybrid Hcomp coding, which allows the EVS 11 to output the event data with a higher compression rate.

[0062] <Application Examples of Hybrid Hcomp Coding> Application examples of Hybrid Hcomp coding will be described with reference to FIGS.

[0063] 7 shows an example of a configuration in which the compressed data (indicator, X-direction address, and event polarity) output from the compression unit 45 is grouped in byte units. For example, by the compression unit 45 outputting the compressed data in byte units, processing at a stage subsequent to the compression unit 45 can be easily performed.

[0064] For example, as shown in A of FIG. 7, when the compression unit 45 compresses event data using the absolute value of the address in the X direction (11 bits), it outputs compressed data in byte units (16 bits) consisting of an indicator (2 bits), a reserve (2 bits), the absolute value of the address in the X direction (11 bits), and the polarity of the event (1 bit).

[0065] 7B, when the compression unit 45 compresses event data using the relative value of the X-direction address (6 bits), it outputs compressed data in byte units (16 bits) consisting of an indicator (2 bits), the relative value of the X-direction address (6 bits), the polarity of the event (1 bit), the relative value of the X-direction address (6 bits), and the polarity of the event (1 bit). In this case, the compressed data in byte units is made up of two event units and corresponds to continuous event data within a 6-bit width.

[0066] 7C, when the compression unit 45 compresses event data using the relative value of the X-direction address (3 bits), it outputs compressed data in byte units (16 bits) consisting of an indicator (2 bits), a reserve (2 bits), the relative value of the X-direction address (3 bits), the polarity of the event (1 bit), the relative value of the X-direction address (3 bits), the polarity of the event (1 bit), the relative value of the X-direction address (3 bits), and the polarity of the event (1 bit). In this case, the compressed data in byte units is made up of three event units and corresponds to continuous event data within a 3-bit width.

[0067] 7D, when compressing event data using the relative value of the X-direction address (2 bits), the compression unit 45 outputs compressed data in bytes (16 bits) consisting of an indicator (2 bits), a reserve (2 bits), the relative value of the X-direction address (2 bits), the polarity of the event (1 bit), the relative value of the X-direction address (2 bits), the polarity of the event (1 bit), the relative value of the X-direction address (2 bits), the polarity of the event (1 bit), the relative value of the X-direction address (2 bits), the polarity of the event (1 bit), the relative value of the X-direction address (2 bits), and the polarity of the event (1 bit). In this case, the compressed data in bytes is made up of four event units and corresponds to continuous event data within a 2-bit width.

[0068] The bit allocation in the event unit is not limited to the example shown in Fig. 7 (i.e., the order in which the polarity of the event is allocated to the lower bits of the event unit and the address in the X direction is allocated to the upper bits of the event unit). Also, the indicator may have polarity information for each event.

[0069] For example, as shown in A of FIG. 8, when compressing event data using the absolute value of the address in the X direction (11 bits), the compression unit 45 may assign bits in the event unit in the following order: indicator (2 bits), reserve (2 bits), event polarity (1 bit), and absolute value of the address in the X direction (11 bits).

[0070] Similarly, as shown in B of FIG. 8, when compressing event data using the relative value of the address in the X direction (6 bits), the compression unit 45 may assign bits in the event unit in the following order: indicator (2 bits), event polarity (1 bit), relative value of the address in the X direction (6 bits), event polarity (1 bit), and relative value of the address in the X direction (6 bits).

[0071] When compressing event data using the relative value of the address in the X direction (6 bits), the compressor 45 may allocate bits in the event unit symmetrically between the upper 8 bits and the lower 8 bits, taking into consideration ease of decoding (reduction of computational load). That is, as shown in C of Fig. 8, the compressor 45 may allocate bits in the event unit in the following order: indicator (1 bit), event polarity (1 bit), relative value of the address in the X direction (6 bits), reserve (1 bit), event polarity (1 bit), and relative value of the address in the X direction (6 bits).

[0072] Furthermore, the compressor 45 may compress the event data so that absolute and relative values ​​of the X-direction address are mixed. For example, as shown in D of Fig. 8, the compressor 45 may assign bits in the event unit so that absolute and relative values ​​are mixed, such as an indicator (2 bits), event polarity (1 bit), relative value of the X-direction address (4 bits), event polarity (1 bit), and absolute value of the X-direction address (8 bits).

[0073] With reference to FIG. 9, application of the encoding method of the present technology to addresses in the Y direction will be described.

[0074] For example, the encoding method of the present technology can be applied not only to addresses in the X direction as described above, but also to addresses in the Y direction stored in packet headers, event information, etc. In other words, just as the EVS 11 compresses event data by changing the expression method of the encoding method within one line (i.e., the absolute value or relative value of the address in the X direction) on an event data unit basis, the EVS 11 can reduce the amount of data in packet headers by changing the absolute value or relative value of the address in the Y direction on a line unit basis.

[0075] For example, as shown in the upper part of Fig. 9, conventionally, the absolute value of the address in the Y direction, Y_address (14 bits), was always stored in the packet header regardless of the density of the line spacing within one frame. In contrast, as shown in the lower part of Fig. 9, the EVS11 stores the absolute value of the address in the Y direction, Y_address (14 bits), in the packet header according to the density of the line spacing within one frame when the line spacing is sparse, and stores the relative value of the address in the Y direction, Y_diff (5 bits) in the packet header when the line spacing is dense, thereby making it possible to reduce the amount of data in the packet header.

[0076] In this case, an indicator Y_Indicator is stored in the packet header to identify for each packet whether an absolute value or a relative value is used for the Y-direction address. The relative value Y_diff of the Y-direction address is a differential value from the Y-direction address on the previous line. While an example of five bits is shown in FIG. 9, other bit counts are also acceptable. For example, the number of bits of the indicator Y_Indicator may be increased so that multiple bit counts can be selected for the relative value Y_diff of the Y-direction address.

[0077] 9 is an example, and the present invention is not limited to such a structure. Also, the indicator Y_Indicator may be configured to be included in a packet indicator, which is a header identifier (e.g., a frame header, a line header, etc.).

[0078] 10A and 10B are diagrams illustrating application of the encoding method of the present technology to timestamps.

[0079] For example, the encoding method of the present technology can be applied to the address in the X direction as described above, and also to the timestamp (time information) indicating the time when an event occurred. In other words, the EVS 11 can reduce the amount of data by changing the absolute value or relative value of the timestamp in timestamp units, just as the EVS 11 compresses event data by changing the expression method of the encoding method within one line (i.e., the absolute value or relative value of the address in the X direction).

[0080] For example, as shown in the upper part of Fig. 10, conventionally, an absolute timestamp (16 bits) was always stored in the packet header regardless of the spacing between timestamps within a frame. In contrast, as shown in the lower part of Fig. 10, the EVS 11 stores the absolute timestamp (16 bits) in the packet header according to the spacing between timestamps, if the spacing between timestamps is sparse, and stores the relative timestamp value T_diff (7 bits) in the packet header if the spacing between timestamps is dense, thereby reducing the amount of data in the packet header.

[0081] In this case, an indicator T_Indicator is stored in the packet header to identify whether the timestamp is an absolute value or a relative value for each packet. The relative value T_diff of the timestamp is the difference from the previous timestamp, and although an example of 7 bits is shown in Figure 10, other bit counts are also acceptable.

[0082] The structures of the packet header and event unit shown in Figure 10 are merely examples, and the present invention is not limited to these structures. For example, the timestamp may be stored in a location other than the packet header, or may be stored in the event unit. Also, the timestamp for one frame may be stored in the frame header.

[0083] As described above, the encoding method of the present technology can be applied to the X-direction address, the Y-direction address, and the timestamp individually. Furthermore, the encoding method of the present technology may be applied to the X-direction address, the Y-direction address, and the timestamp in combination.

[0084] <Input Data and Output Data> Input data input to the compression unit 45 and output data output from the compression unit 45 will be described with reference to FIGS.

[0085] FIG. 11 shows an example of a data format used for the input data input to the compression unit 45.

[0086] 11A, the input data may be in a data format consisting of event data arranged in chronological order according to the timing at which the event was detected. For example, the event data may include addresses x and y of EVS pixels (pixels for event detection) at which the event was detected, polarity information p indicating the polarity of the event (positive event or negative event), and a timestamp t indicating the time at which the event was detected.

[0087] 11B, the input data may be in the form of a count map, which is configured by mapping the number of events for each EVS pixel detected per unit time (e.g., one frame period) according to the arrangement of the EVS pixels. Note that the number of events may be counted by distinguishing between the occurrence of positive events and the occurrence of negative events, or may be counted without distinguishing between the occurrence of positive events and the occurrence of negative events.

[0088] As shown in FIG. 11C , the input data may use an event frame data format in which pixel values ​​indicating the polarity of an event are arranged according to the arrangement of EVS pixels for each frame period. For example, the input data may be represented by a two-bit pixel value (P=11′) indicating that a positive event has been detected, a two-bit pixel value (N=10′) indicating that a negative event has been detected, or a two-bit pixel value (0=00′) indicating that no event has been detected. Note that if both a positive event and a negative event are detected in the same EVS pixel during one frame period, the polarity of the last detected event may be used as the pixel value of that EVS pixel.

[0089] 11D, the input data may use an event address data format that includes, for each frame period, the address x and address y of an EVS pixel where an event is detected, and polarity information p (positive event or negative event) indicating the polarity of the event. Note that for EVS pixels where no event is detected in one frame period, the input data contains no data.

[0090] FIG. 12 shows an example of output data when Hcomp coding is used as the encoding method for compressing input data.

[0091] In Hcomp coding, input data is divided into the address x of the EVS pixel where the i-th event occurred. ｉ , and the polarity p of the i-th event ｉ This is a method of encoding into output data expressed as (positive event: 0, negative event: 1).

[0092] For example, as shown in the figure, a case will be described in which one line of input data is coded by Hcomp coding, in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels. In this case, following the address y of that line, output data (x ０ , p０ ) = (0, 0) is output. Similarly, output data (x １ , p １ ) = (11, 1), the output data (x ２ , p ２ ) = (12, 1), the output data (x ３ , p ３ )=(13, 1) is output. When outputting the output data of all lines, the line address y is not required.

[0093] FIG. 13 shows an example of output data when run length coding is used as the encoding method for compressing input data.

[0094] In Run Length Coding, input data is divided into the i-th data type d ｉ (positive event: 1, negative event: 2, no event: 0), and the number of consecutive data c ｉ This is a method of encoding data into output data expressed as

[0095] For example, as shown in the figure, a case will be described in which one line of input data is coded by run length coding, in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels. In this case, the address y of that line is followed by output data (d ０ , c ０ ) = (1, 1) is output. Similarly, output data (d １ , c １ ) = (0, 10), and the output data (d ２ , c ２ ) = (2, 3), output data (d ３ , c３ )=(0, 2) is output. When outputting the output data of all lines, the line address y is not required.

[0096] FIG. 14 shows an example of output data when Huffman coding is used as the encoding method for compressing input data.

[0097] In Huffman coding, input data is divided into the i-th coded data h according to the frequency of the number of events (number of data occurrences). ｉ , and output data represented by a frequency table for decoding. Note that one frequency table is provided for each compression unit.

[0098] For example, as shown in the figure, a case will be described in which one line of input data is coded by Huffman coding, in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels. In this case, the address y of that line is followed by the 0th to 5th coded data (h ０ , h １ , h ２ , h ３ , h ４ , h ５ ) = (7, 2, 0, 0, 0, 6) is output, and the frequency table shown in the figure is output. The frequency table contains the following registered information: the Huffman code indicating that the number of occurrences of negative event (= 2) data is 3 is 0 (= 0), the Huffman code indicating that the number of occurrences of data with 10 consecutive no events is 1 is 10 (= 2), the Huffman code indicating that the number of occurrences of data with two consecutive no events is 1 is 100 (= 6), and the Huffman code indicating that the number of occurrences of data with positive event (= 1) is 111 (= 7). Note that when output data for all lines is output, the line address y is not required.

[0099] FIG. 15 shows an example of output data when Light Huffman coding is used as the encoding method for compressing input data.

[0100] Light Huffman coding is a method of encoding input data into output data in which no event is represented by one bit and the presence of an event is represented by two bits.

[0101] For example, as shown in the figure, all events in the input data are represented by two bits (positive event: 01, negative event: 10, no event: 00). When such input data is encoded using Light Huffman coding, the output data represents events by two bits (positive event: 11, negative event: 10) and no event by one bit (:0).

[0102] FIG. 16 shows an example of output data when Event Distance coding is used as the encoding method for compressing input data.

[0103] Event Distance coding is a method of coding input data based on the distance d from the previous event of the i-th event. ｉ , polarity p of the i-th event ｉ (positive event: 0, negative event: 1), and the number of consecutive i-th data c ｉ This is a method of encoding data into output data expressed as

[0104] For example, as shown in the figure, a case will be described in which one line of input data in which a positive event is detected in the 0th EVS pixel, no event is detected in the 1st to 10th EVS pixels, a negative event is detected in the 11th to 13th EVS pixels, and no event is detected in the 14th and 15th EVS pixels is coded using Event Distance coding. In this case, following the address y of that line, output data (d ０ , p ０ , c ０,) = (0,0,1) is output. Similarly, output data (d １ , p １ , c １ ,)=(10,1,3) is output. When outputting the output data of all lines, the line address y is not required.

[0105] In addition to the encoding methods described with reference to FIGS. 12 to 16 , the present technology can employ various encoding methods, such as the encoding method according to ISSCC (IEEE International Solid-State Circuits Conference) 2017 4.1 and the encoding method according to ISSCC2020 5.10.

[0106] The output data of the metadata generated from the input data will be described with reference to FIGS.

[0107] FIG. 17 shows an example of output data when an optical flow that represents the movement of an object on an image as a vector is generated from input data.

[0108] Optical flow generation is performed by calculating the address x of the i-th block in the x direction from the input data. ｉ , the y-direction address y of the i-th block ｉ , the optical flow in the x direction of the i-th block x_dis ｉ , the optical flow in the y direction of the i-th block, y_dis ｉ , and the acquisition time of the i-th optical flow t ｉ When outputting all blocks, the address x ｉ and address y ｉ is not required, and the acquisition time t ｉ is not necessarily required.

[0109] For example, input data in any of the data formats shown in FIG. 11 (input data in the data format shown in A of FIG. 11 (x ｉ , y ｉ , pｉ , t ｉ ) is shown in the figure), frame data is reconstructed from input data input during one frame period, and optical flow is calculated, for example, in units of blocks, from frame data of one or more previous frames previously reconstructed and frame data of the current frame reconstructed this time, and output data (x ｉ , y ｉ , x_dis ｉ , y_dis ｉ , t ｉ ) is output. For example, the optical flow may include the amount of movement and the speed of movement of each block. Note that a block may be a single pixel, or may be an area of ​​M×N pixels (both M and N are integers equal to or greater than 1).

[0110] FIG. 18 shows an example of output data when feature points representing characteristic points on an image are extracted from input data.

[0111] Feature point extraction is performed by finding the address x of the i-th feature point in the x direction from the input data. ｉ , the y-direction address of the i-th feature point y ｉ , and the extraction time t of the i-th feature point ｉ The extraction time of the i-th feature point is t ｉ is not necessarily required.

[0112] For example, input data in any of the data formats shown in FIG. 11 (input data in the data format shown in A of FIG. 11 (x ｉ , y ｉ , p ｉ , t ｉ ) is input, frame data is reconstructed from input data input during one frame period, feature points are extracted from the reconstructed frame data, and output data (x ｉ , y ｉ , t ｉ ) is output.

[0113] FIG. 19 shows an example of output data when an ROI (Region Of Interest) indicating a specific region of interest to be processed on an image is extracted from input data.

[0114] The ROI extraction is performed by extracting the ROI coordinate x_sta in the x direction, which indicates the start position of the ROI for the i-th frame, from the input data. ｉ , the ROI coordinate x_end in the x direction indicating the end position of the ROI for the i-th frame ｉ , the ROI coordinate y_sta in the y direction indicating the start position of the ROI for the i-th frame ｉ , y_end, the ROI coordinate in the y direction indicating the end position of the ROI for the i-th frame ｉ , and the extraction time of the ROI for the i-th frame t ｉ The ROI extraction time for the i-th frame is t ｉ is not necessarily required.

[0115] For example, input data in any of the data formats shown in FIG. 11 (input data in the data format shown in A of FIG. 11 (x ｉ , y ｉ , p ｉ , t ｉ ) is input, frame data is reconstructed from input data input during one frame period, and a process of extracting an ROI from the reconstructed frame data is performed, and output data (x_sta ｉ , x_end ｉ , y_sta ｉ , y_end ｉ , t ｉ ) is output. In addition to the output data from which the ROI has been extracted, or instead of the output data from which the ROI has been extracted, image data cut out as the ROI from the frame data may be output as shown in FIG.

[0116] In addition to a configuration in which the EVS 11 is mounted alone on a sensor chip, a configuration in which the EVS 11 is mounted on a sensor chip together with a CMOS image sensor having RGB pixels for capturing color images may also be adopted. For example, in a configuration in which the EVS 11 and the CMOS image sensor are mounted on a single sensor chip, various configuration examples can be adopted, such as a configuration example in which the EVS pixels and the RGB pixels are arranged at a uniform density overall, or a configuration example in which the EVS pixels and the RGB pixels are arranged by dividing them into positions, as disclosed in International Publication No. 2023 / 058670.

[0117] <Example of Use of Image Sensor> FIG. 21 is a diagram showing an example of use of the image sensor (EVS) described above.

[0118] The image sensor described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0119] - Devices for taking images for viewing, such as digital cameras and mobile devices with camera functions. - Devices for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. - Devices for home appliances such as TVs, refrigerators, and air conditioners that take images of the user's gestures and operate the device according to those gestures. - Devices for medical and healthcare purposes, such as endoscopes and devices that take images of blood vessels by receiving infrared light. - Devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. - Devices for beauty purposes, such as skin measuring devices that take images of the skin and microscopes that take images of the scalp. - Devices for sports, such as action cameras and wearable cameras for sports. - Devices for agriculture, such as cameras to monitor the condition of fields and crops.

[0120] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) A data processing device comprising: an event detection unit that detects the occurrence of an event based on a change in luminance of light received by a plurality of event detection pixels and generates event data indicating the details of the event; and a compression unit that compresses the event data by changing a representation method of an encoding format for encoding the event data based on an occurrence density of the event data. (2) The data processing device according to (1), wherein the compression unit compresses the event data using the encoding format that has a high affinity for both high and low occurrence densities of the event data. (3) The data processing device according to (2), wherein the compression unit encodes the event data for event data with a high occurrence density using the encoding format of the representation method that has a high compression rate when the occurrence density is high, and encodes the event data for event data with a low occurrence density using the encoding format of the representation method that has a high compression rate when the occurrence density is low. (4) The data processing device according to (3), wherein the compression unit changes the X-direction address of the event data within one line between an absolute value and a relative value for each piece of event data based on the local occurrence density of the event data within one line. (5) The data processing device according to (4), wherein the compression unit sets an indicator for identifying whether an absolute value or a relative value is used as the X-direction address of the event data. (6) The data processing device according to (4) or (5), wherein the compression unit encodes the event data having a low occurrence density within one line using the absolute value of the X-direction address of the event data, and encodes the event data having a high occurrence density within one line using the relative value of the X-direction address of the event data. (7) The data processing device according to any of (4) to (6), wherein the compression unit determines the occurrence density of the event data according to a threshold value for an address difference, which is the difference value from the X-direction address of the immediately preceding event data.(8) The data processing device according to (7), wherein the compression unit encodes the event data using an absolute value of the X-direction address of the event data when the address difference is greater than the threshold, and encodes the event data using a relative value of the X-direction address of the event data when the address difference is equal to or less than the threshold. (9) The data processing device according to (7) or (8), wherein the threshold is set according to the number of bits used as the relative value of the X-direction address. (10) The data processing device according to (9), wherein the threshold is set to 7 when the number of bits used as the relative value of the X-direction address is 3. (11) The data processing device according to any of (4) to (10), wherein the compression unit groups the event data into compressed data in byte units and outputs the compressed data. (12) The data processing device according to (11), wherein the compression unit assigns the X-direction address of the event to the most significant bits of an event unit formed by the X-direction address of the event and the polarity of the event, and assigns the polarity of the event to the least significant bits of the event unit. (13) The data processing device according to (11) or (12), wherein, when two event units each consisting of an X-direction address of the event and a polarity of the event are stored in one byte of the compressed data, the compressed data is configured so that the upper and lower bits of the compressed data are symmetrical. (14) The data processing device according to any of (11) to (13), wherein the compressed data is configured so that absolute and relative values ​​of the X-direction address of the event are mixed in one byte of the compressed data. (15) The data processing device according to any of (4) to (14), wherein the absolute or relative value of the Y-direction address stored in a packet header of a packet in which the event data is stored is changed in units of lines in accordance with the density of line spacing within one frame. (16) The data processing device according to any of (4) to (15), wherein the absolute or relative value of time information indicating the time when the event occurred is changed in units of the time information.(17) A data processing method including: a data processing device detecting the occurrence of an event based on a change in luminance of light received by a plurality of event detection pixels, and generating event data indicating the details of the event; and compressing the event data by changing the representation method of an encoding format for encoding the event data based on the occurrence density of the event data. (18) An event data output sensor including: a sensor unit in which a plurality of event detection pixels are arranged in an array in a pixel area; an event detection unit that detects the occurrence of an event based on a change in luminance of light received by the plurality of event detection pixels, and generates event data indicating the details of the event; and a compression unit that compresses the event data by changing the representation method of an encoding format for encoding the event data based on the occurrence density of the event data.

[0121] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0122] REFERENCE SIGNS LIST 11 EVS, 21 Sensor unit, 22 Event detection unit, 23 Event information generation processing unit, 24 Packet generation processing unit, 25 Data output unit, 31 Pixel area, 32 Line control unit, 33 Column control unit, 41 Event / line count unit, 42 Event count unit, 43 Event number analysis unit, 44 Data amount calculation unit, 45 Compression unit, 51 Header generation unit, 52 Packet generation unit

Claims

1. A data processing device comprising: an event detection unit that detects the occurrence of an event based on changes in the brightness of light received by multiple event detection pixels and generates event data indicating the content of the event; and a compression unit that compresses the event data by changing the representation method of the encoding method that encodes the event data based on the occurrence density of the event data.

2. The data processing device according to claim 1, wherein the compression unit compresses the event data using the encoding method that has a high compatibility with both the density of occurrence of the event data and the frequency of occurrence of the event data.

3. The data processing device according to claim 2, wherein the compression unit encodes the event data for event data with a high occurrence density using the encoding method of the representation method that has a high compression rate when the occurrence density is high, and encodes the event data for event data with a low occurrence density using the encoding method of the representation method that has a high compression rate when the occurrence density is low.

4. A data processing device according to claim 3, wherein the compression unit changes the X-direction address of the event data within one line between absolute and relative values ​​for each event data unit based on the local density of occurrence of the event data within one line.

5. A data processing device according to claim 4, wherein said compression section sets an indicator for identifying whether an absolute value or a relative value is used as the address in the X direction of said event data.

6. A data processing device according to claim 4, wherein the compression unit encodes the event data that occurs at a low density within one line using the absolute value of the X-direction address of the event data, and encodes the event data that occurs at a high density within one line using the relative value of the X-direction address of the event data.

7. A data processing device according to claim 4, wherein the compression section determines the occurrence density of the event data in accordance with a threshold value for an address difference, which is a difference value from the address of the immediately preceding event data in the X direction.

8. The data processing device according to claim 7, wherein the compression unit encodes the event data using an absolute value of the address in the X direction of the event data when the address difference is greater than the threshold value, and encodes the event data using a relative value of the address in the X direction of the event data when the address difference is equal to or less than the threshold value.

9. A data processing device according to claim 7, wherein the threshold value is set in accordance with the number of bits used as the relative value of the address in the X direction.

10. A data processing device according to claim 9, wherein the threshold value is set to 7 when the number of bits used as the relative value of the address in the X direction is 3 bits.

11. The data processing device according to claim 4, wherein the compression section compresses the event data and outputs the compressed data in units of bytes.

12. A data processing device according to claim 11, wherein the compression section assigns the X-direction address of the event to the most significant bits of an event unit formed by the X-direction address of the event and the polarity of the event, and assigns the polarity of the event to the least significant bits of the event unit.

13. A data processing device according to claim 11, wherein the compressed data is configured so that when two event units, each consisting of the X-direction address of the event and the polarity of the event, are stored in one byte of the compressed data, the upper and lower bits of the compressed data are symmetrical.

14. The data processing device according to claim 11, wherein the compressed data is configured so that absolute and relative values ​​of the X-direction addresses of the events are mixed in one byte of the compressed data.

15. A data processing device according to claim 4, wherein the absolute or relative value of the Y-direction address stored in the packet header of the packet in which the event data is stored is changed in units of said line according to the density of the line spacing within one frame.

16. A data processing device according to claim 4, wherein the absolute value or relative value of the time information is changed in units of the time information according to the fineness of the time information indicating the time when the event occurred.

17. A data processing method comprising: a data processing device detecting the occurrence of an event based on a change in the luminance of light received by a plurality of event detection pixels, generating event data indicating the content of the event, and compressing the event data by changing the representation method of an encoding method that encodes the event data based on the occurrence density of the event data.

18. An event data output sensor comprising: a sensor unit in which a plurality of event detection pixels are arranged in an array in a pixel area; an event detection unit that detects the occurrence of an event based on a change in the luminance of light received by the plurality of event detection pixels and generates event data indicating the content of the event; and a compression unit that compresses the event data by changing the expression method of the encoding method that encodes the event data based on the occurrence density of the event data.

Images